[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:13 dist-test-slave-k1dg krb5kdc[21184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:13 dist-test-slave-k1dg krb5kdc[21184](info): set up 2 sockets
Sep 29 03:10:13 dist-test-slave-k1dg krb5kdc[21184](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 03:10:14.594362 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:14 dist-test-slave-k1dg krb5kdc[21184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:14.735332 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:35613
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:35613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:14.851531 21197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:14.851825 21197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:14.851920 21197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:14.856343 21197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:14.856424 21197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:14.856446 21197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:14.856462 21197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:14.856478 21197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:14.862241 21197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:35613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:35613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:14.863447 21197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:14.864601 21197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:14.874689 21202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:14.874675 21203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:14.874733 21197 server_base.cc:1034] running on GCE node
W20260929 03:10:14.874676 21205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:14.875626 21197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:14.876773 21197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:14.877985 21197 hybrid_clock.cc:648] HybridClock initialized: now 1790651414877974 us; error 35 us; skew 500 ppm
Sep 29 03:10:14 dist-test-slave-k1dg krb5kdc[21184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651414, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:14.888063 21197 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:14.889109 21197 webserver.cc:460] Webserver started at http://127.20.174.126:43061/ using document root <none> and password file <none>
I20260929 03:10:14.889760 21197 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:14.889864 21197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:14.890116 21197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:14.892359 21197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "635e84d03b094632832a774cf9744b68"
format_stamp: "Formatted at 2026-09-29 03:10:14 on dist-test-slave-k1dg"
server_key: "2ba3ba7d6bda9cf6dabcca99f9c770ca"
server_key_iv: "e6aa9e2d6ffdea4b2963d208c988f78f"
server_key_version: "encryptionkey@0"
I20260929 03:10:14.892982 21197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "635e84d03b094632832a774cf9744b68"
format_stamp: "Formatted at 2026-09-29 03:10:14 on dist-test-slave-k1dg"
server_key: "2ba3ba7d6bda9cf6dabcca99f9c770ca"
server_key_iv: "e6aa9e2d6ffdea4b2963d208c988f78f"
server_key_version: "encryptionkey@0"
I20260929 03:10:14.896912 21197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:14.899708 21212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:14.901046 21197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:14.901211 21197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "635e84d03b094632832a774cf9744b68"
format_stamp: "Formatted at 2026-09-29 03:10:14 on dist-test-slave-k1dg"
server_key: "2ba3ba7d6bda9cf6dabcca99f9c770ca"
server_key_iv: "e6aa9e2d6ffdea4b2963d208c988f78f"
server_key_version: "encryptionkey@0"
I20260929 03:10:14.901343 21197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:14.916492 21197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:14.919706 21197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:14.919919 21197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:14.927325 21197 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:35613
I20260929 03:10:14.927345 21264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:35613 every 8 connection(s)
I20260929 03:10:14.928346 21197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:14.931424 21265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:14.932667 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 21197
I20260929 03:10:14.932837 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:14.934284 21177 external_mini_cluster.cc:1428] Setting key 0189905741f0b6dcf096e0b3d3ed5ae0
I20260929 03:10:14.937742 21265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68: Bootstrap starting.
I20260929 03:10:14.940332 21265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:14.941107 21265 log.cc:824] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:14.943164 21265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68: No bootstrap required, opened a new log
I20260929 03:10:14.946728 21265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e84d03b094632832a774cf9744b68" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 35613 } }
I20260929 03:10:14.946987 21265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:14.947065 21265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635e84d03b094632832a774cf9744b68, State: Initialized, Role: FOLLOWER
I20260929 03:10:14.947499 21265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e84d03b094632832a774cf9744b68" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 35613 } }
I20260929 03:10:14.947645 21265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:14.947747 21265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:14.947871 21265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:14.948792 21265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e84d03b094632832a774cf9744b68" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 35613 } }
I20260929 03:10:14.949134 21265 leader_election.cc:302] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635e84d03b094632832a774cf9744b68; no voters:
I20260929 03:10:14.949438 21265 leader_election.cc:288] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:14.949563 21270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:14.949774 21270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [term 1 LEADER]: Becoming Leader. State: Replica: 635e84d03b094632832a774cf9744b68, State: Running, Role: LEADER
I20260929 03:10:14.950119 21270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e84d03b094632832a774cf9744b68" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 35613 } }
I20260929 03:10:14.950846 21265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:14.951355 21272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635e84d03b094632832a774cf9744b68. Latest consensus state: current_term: 1 leader_uuid: "635e84d03b094632832a774cf9744b68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e84d03b094632832a774cf9744b68" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 35613 } } }
I20260929 03:10:14.951485 21272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:14.951812 21271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "635e84d03b094632832a774cf9744b68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635e84d03b094632832a774cf9744b68" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 35613 } } }
I20260929 03:10:14.951897 21271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68 [sys.catalog]: This master's current role is: LEADER
Sep 29 03:10:14 dist-test-slave-k1dg krb5kdc[21184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
W20260929 03:10:14.956269 21285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:10:14.956377 21285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:10:14.956548 21274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:14.957458 21274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:14.960155 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:14.937646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.20.174.126:35613)
0929 03:10:14.938124 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:14.938139 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:14.939227 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 03:10:14.942606 (+ 3379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:14.942623 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:14.942945 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:14.943282 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:14.945477 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:14.946512 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:14.947984 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:14.948456 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:14.948636 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:14.954165 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:14.954177 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:14.954189 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:14.954215 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:14.956374 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:14.957045 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:14.957050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:14.957054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:14.957167 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:14.957594 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:14.957599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:14.957601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:14.958009 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:14.958220 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:14.958895 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 03:10:14.959176 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":168,"threads_started":1}
I20260929 03:10:14.964311 21274 catalog_manager.cc:1347] Generated new cluster ID: b7d539ad737447a4983fefe8be705aef
I20260929 03:10:14.964409 21274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:14.984153 21274 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:14.985504 21274 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:14.995596 21274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 635e84d03b094632832a774cf9744b68: Generated new TSK 0
I20260929 03:10:14.996565 21274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:10:15.034731 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:35613
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:15.172040 21291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:15.172380 21291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:15.172470 21291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:15.176626 21291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:15.176741 21291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:15.176841 21291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:15.182137 21291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:35613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:15.183251 21291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:15.184294 21291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:15.194301 21297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:15.194309 21299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:15.194504 21291 server_base.cc:1034] running on GCE node
W20260929 03:10:15.194309 21296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:15.195281 21291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:15.195943 21291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:15.197160 21291 hybrid_clock.cc:648] HybridClock initialized: now 1790651415197123 us; error 50 us; skew 500 ppm
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651415, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:15.200537 21291 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:10:15.201632 21291 webserver.cc:460] Webserver started at http://127.20.174.65:44541/ using document root <none> and password file <none>
I20260929 03:10:15.202311 21291 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:15.202427 21291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:15.202723 21291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:15.205444 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "e61db618e0e1480facb07aa75208bcc1"
format_stamp: "Formatted at 2026-09-29 03:10:15 on dist-test-slave-k1dg"
server_key: "2d16b2c1a05c3c176bb0a86edb517545"
server_key_iv: "5fa198fd60e04d9ba51678ab474f7f9b"
server_key_version: "encryptionkey@0"
I20260929 03:10:15.206214 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "e61db618e0e1480facb07aa75208bcc1"
format_stamp: "Formatted at 2026-09-29 03:10:15 on dist-test-slave-k1dg"
server_key: "2d16b2c1a05c3c176bb0a86edb517545"
server_key_iv: "5fa198fd60e04d9ba51678ab474f7f9b"
server_key_version: "encryptionkey@0"
I20260929 03:10:15.211017 21291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:10:15.213783 21306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:15.215288 21291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:15.215461 21291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "e61db618e0e1480facb07aa75208bcc1"
format_stamp: "Formatted at 2026-09-29 03:10:15 on dist-test-slave-k1dg"
server_key: "2d16b2c1a05c3c176bb0a86edb517545"
server_key_iv: "5fa198fd60e04d9ba51678ab474f7f9b"
server_key_version: "encryptionkey@0"
I20260929 03:10:15.215602 21291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:15.251969 21291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:15.256016 21291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:15.256259 21291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:15.256944 21291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:15.257907 21291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:15.257990 21291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:15.258072 21291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:15.258124 21291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:15.268739 21291 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:40545
I20260929 03:10:15.268777 21419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:40545 every 8 connection(s)
I20260929 03:10:15.270022 21291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:15.274515 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 21291
I20260929 03:10:15.274631 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:15.274924 21177 external_mini_cluster.cc:1428] Setting key 073c98eb8a76163d419a8244f17b5f6f
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651415, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:15.284664 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:15.272038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:50263 (local address 127.20.174.126:35613)
0929 03:10:15.272228 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:15.272234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:15.273458 (+ 1224us) server_negotiation.cc:407] Connection header received
0929 03:10:15.274312 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:15.274314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:15.274370 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:15.274524 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:15.275934 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.276460 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.277617 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.277859 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.277976 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:15.280713 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:15.280716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:15.280718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:15.280752 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:15.282425 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:15.283041 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:15.283046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:15.283048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:15.283136 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:15.283533 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:15.283538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:15.283539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:15.283729 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:15.283838 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:15.284333 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 03:10:15.284483 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:10:15.285493 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:15.272486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:35613 (local address 127.20.174.65:50263)
0929 03:10:15.273230 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:15.273282 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:15.274081 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:15.274706 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:15.274716 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:15.275217 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:15.275773 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:15.275787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.276606 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.276609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:15.277488 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:15.277497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.278016 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.278020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:15.278186 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:15.278887 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:15.278904 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:15.280513 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:15.282624 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:15.282630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:15.282646 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:15.282922 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:15.283277 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:15.283280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:15.283282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:15.283433 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:15.283835 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:15.283843 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:15.284093 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 03:10:15.284282 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 03:10:15.284636 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":124,"threads_started":1}
I20260929 03:10:15.286825 21420 heartbeater.cc:347] Connected to a master server at 127.20.174.126:35613
I20260929 03:10:15.287056 21420 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:15.287561 21420 heartbeater.cc:511] Master 127.20.174.126:35613 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:10:15.289247 21229 ts_manager.cc:194] Registered new tserver with Master: e61db618e0e1480facb07aa75208bcc1 (127.20.174.65:40545)
I20260929 03:10:15.290560 21229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:50263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:10:15.307120 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:35613
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:15.425354 21425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:15.425624 21425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:15.425674 21425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:15.429795 21425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:15.429863 21425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:15.429924 21425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:15.435425 21425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:35613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:15.436487 21425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:15.437477 21425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:15.447824 21433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:15.447883 21430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:15.447919 21425 server_base.cc:1034] running on GCE node
W20260929 03:10:15.448011 21431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:15.448652 21425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:15.449347 21425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:15.450558 21425 hybrid_clock.cc:648] HybridClock initialized: now 1790651415450522 us; error 57 us; skew 500 ppm
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651415, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:15.453910 21425 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:10:15.454998 21425 webserver.cc:460] Webserver started at http://127.20.174.66:36195/ using document root <none> and password file <none>
I20260929 03:10:15.455659 21425 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:15.455734 21425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:15.456001 21425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:15.458287 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "d250802f37f144c39117d176eaf01d80"
format_stamp: "Formatted at 2026-09-29 03:10:15 on dist-test-slave-k1dg"
server_key: "ee7cc94214bbd7f3dd0203017fb5723f"
server_key_iv: "1f546366421019adab45d95df5dcacba"
server_key_version: "encryptionkey@0"
I20260929 03:10:15.459007 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "d250802f37f144c39117d176eaf01d80"
format_stamp: "Formatted at 2026-09-29 03:10:15 on dist-test-slave-k1dg"
server_key: "ee7cc94214bbd7f3dd0203017fb5723f"
server_key_iv: "1f546366421019adab45d95df5dcacba"
server_key_version: "encryptionkey@0"
I20260929 03:10:15.463481 21425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:10:15.466364 21440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:15.467985 21425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:10:15.468235 21425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "d250802f37f144c39117d176eaf01d80"
format_stamp: "Formatted at 2026-09-29 03:10:15 on dist-test-slave-k1dg"
server_key: "ee7cc94214bbd7f3dd0203017fb5723f"
server_key_iv: "1f546366421019adab45d95df5dcacba"
server_key_version: "encryptionkey@0"
I20260929 03:10:15.468374 21425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:15.499779 21425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:15.503952 21425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:15.504187 21425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:15.504923 21425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:15.505968 21425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:15.506035 21425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:15.506083 21425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:15.506100 21425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:15.516408 21425 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:35567
I20260929 03:10:15.516430 21553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:35567 every 8 connection(s)
I20260929 03:10:15.517571 21425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:15.524209 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 21425
I20260929 03:10:15.524369 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:15.524616 21177 external_mini_cluster.cc:1428] Setting key c456e3683e91fdd9f728292b559f5815
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651415, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:15.533187 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:15.519669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:56957 (local address 127.20.174.126:35613)
0929 03:10:15.519818 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:15.519822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:15.521092 (+ 1270us) server_negotiation.cc:407] Connection header received
0929 03:10:15.522116 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:15.522119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:15.522184 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:15.522311 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:15.523682 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.524314 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.525783 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.526114 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.526241 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:15.529052 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:15.529056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:15.529058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:15.529084 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:15.530752 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:15.531377 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:15.531384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:15.531387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:15.531481 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:15.531934 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:15.531937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:15.531938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:15.532144 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:15.532270 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:15.532841 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 03:10:15.532994 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:10:15.534246 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:15.520072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:35613 (local address 127.20.174.66:56957)
0929 03:10:15.520885 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:15.520977 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:15.521864 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:15.522570 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:15.522579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:15.522982 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:15.523512 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:15.523528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.524460 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.524464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:15.525642 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:15.525653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.526284 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.526289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:15.526545 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:15.527210 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:15.527227 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:15.528847 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:15.530897 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:15.530906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:15.530926 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:15.531253 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:15.531588 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:15.531593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:15.531595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:15.531756 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:15.532269 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:15.532278 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:15.532565 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 03:10:15.532778 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 03:10:15.533169 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":126,"threads_started":1}
I20260929 03:10:15.535797 21554 heartbeater.cc:347] Connected to a master server at 127.20.174.126:35613
I20260929 03:10:15.536067 21554 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:15.536613 21554 heartbeater.cc:511] Master 127.20.174.126:35613 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:10:15.537756 21229 ts_manager.cc:194] Registered new tserver with Master: d250802f37f144c39117d176eaf01d80 (127.20.174.66:35567)
I20260929 03:10:15.538395 21229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:56957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:10:15.557731 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:35613
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:15.676884 21559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:15.677160 21559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:15.677238 21559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:15.684134 21559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:15.684270 21559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:15.684392 21559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:15.694069 21559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:35613
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:15.695258 21559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:15.696290 21559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:15.706607 21564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:15.706678 21567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:15.706681 21565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:15.706830 21559 server_base.cc:1034] running on GCE node
I20260929 03:10:15.707396 21559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:15.708145 21559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:15.709339 21559 hybrid_clock.cc:648] HybridClock initialized: now 1790651415709330 us; error 33 us; skew 500 ppm
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651415, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:15.712389 21559 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:10:15.713481 21559 webserver.cc:460] Webserver started at http://127.20.174.67:32955/ using document root <none> and password file <none>
I20260929 03:10:15.714082 21559 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:15.714175 21559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:15.714412 21559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:15.716647 21559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "5ccc1b558fad4ea9991064c18806ee5f"
format_stamp: "Formatted at 2026-09-29 03:10:15 on dist-test-slave-k1dg"
server_key: "7e821a3dc583c5c73ee8c4f9628eafc7"
server_key_iv: "d9c09a57fce69fa28a7bb854f002a550"
server_key_version: "encryptionkey@0"
I20260929 03:10:15.717173 21559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "5ccc1b558fad4ea9991064c18806ee5f"
format_stamp: "Formatted at 2026-09-29 03:10:15 on dist-test-slave-k1dg"
server_key: "7e821a3dc583c5c73ee8c4f9628eafc7"
server_key_iv: "d9c09a57fce69fa28a7bb854f002a550"
server_key_version: "encryptionkey@0"
I20260929 03:10:15.721408 21559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:10:15.724170 21574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:15.725486 21559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:15.725593 21559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "5ccc1b558fad4ea9991064c18806ee5f"
format_stamp: "Formatted at 2026-09-29 03:10:15 on dist-test-slave-k1dg"
server_key: "7e821a3dc583c5c73ee8c4f9628eafc7"
server_key_iv: "d9c09a57fce69fa28a7bb854f002a550"
server_key_version: "encryptionkey@0"
I20260929 03:10:15.725747 21559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:15.739766 21559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:15.743392 21559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:15.743623 21559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:15.744311 21559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:15.745221 21559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:15.745270 21559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:15.745311 21559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:15.745373 21559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:15.755620 21559 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:35835
I20260929 03:10:15.755641 21687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:35835 every 8 connection(s)
I20260929 03:10:15.756868 21559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:15.759043 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 21559
I20260929 03:10:15.759176 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:15.759460 21177 external_mini_cluster.cc:1428] Setting key 54a83017efa9efed14c2eed348a485ed
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651415, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:15.771016 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:15.758972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:39149 (local address 127.20.174.126:35613)
0929 03:10:15.759152 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:15.759159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:15.760391 (+ 1232us) server_negotiation.cc:407] Connection header received
0929 03:10:15.761274 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:15.761278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:15.761330 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:15.761417 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:15.762760 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.763293 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.765081 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.765303 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.765391 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:15.767841 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:15.767844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:15.767846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:15.767873 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:15.769113 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:15.769589 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:15.769591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:15.769592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:15.769669 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:15.769969 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:15.769971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:15.769972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:15.770146 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:15.770223 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:15.770779 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 03:10:15.770887 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:10:15.771922 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:15.759437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:35613 (local address 127.20.174.67:39149)
0929 03:10:15.760227 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:15.760283 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:15.761087 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:15.761547 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:15.761554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:15.762096 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:15.762629 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:15.762645 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.763422 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.763424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:15.764961 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:15.764971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.765450 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.765453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:15.765598 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:15.766125 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:15.766141 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:15.767661 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:15.769217 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:15.769222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:15.769234 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:15.769498 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:15.769750 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:15.769752 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:15.769753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:15.769882 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:15.770230 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:15.770235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:15.770553 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 03:10:15.770732 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 03:10:15.771020 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":150,"threads_started":1}
I20260929 03:10:15.773300 21688 heartbeater.cc:347] Connected to a master server at 127.20.174.126:35613
I20260929 03:10:15.773594 21688 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:15.774186 21688 heartbeater.cc:511] Master 127.20.174.126:35613 requested a full tablet report, sending...
I20260929 03:10:15.775328 21228 ts_manager.cc:194] Registered new tserver with Master: 5ccc1b558fad4ea9991064c18806ee5f (127.20.174.67:35835)
I20260929 03:10:15.775947 21228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:39149
I20260929 03:10:15.785847 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651415, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651415, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:10:15.813663 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:15.803106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.20.174.65:40545)
0929 03:10:15.803434 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:15.803440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:15.803483 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:10:15.803535 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:15.803539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:15.803728 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:15.803853 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:15.804690 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.805537 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.806409 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.806857 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.806990 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:15.809808 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:15.809819 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:15.809837 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:15.809912 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:15.811717 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:15.812313 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:15.812319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:15.812320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:15.812408 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:15.812692 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:15.812695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:15.812696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:15.812979 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:15.813159 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:15.813385 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 03:10:15.813519 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
W20260929 03:10:15.814670 21334 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60278
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:10:15.842670 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:15.833243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.20.174.65:40545)
0929 03:10:15.833430 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:15.833434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:15.833558 (+ 124us) server_negotiation.cc:407] Connection header received
0929 03:10:15.833673 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:15.833675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:15.833720 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:15.833805 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:15.834692 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.835310 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.836207 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:15.836444 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:15.836541 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:15.839312 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:15.839316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:15.839318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:15.839346 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:15.840865 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:15.841391 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:15.841395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:15.841397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:15.841493 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:15.841765 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:15.841768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:15.841769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:15.841962 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:15.842145 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:15.842343 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 03:10:15.842541 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:10:15.843892 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 21291
I20260929 03:10:15.849086 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 21425
I20260929 03:10:15.854045 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 21559
I20260929 03:10:15.859598 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 21197
2026-09-29T03:10:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2348 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21702](info): set up 2 sockets
Sep 29 03:10:15 dist-test-slave-k1dg krb5kdc[21702](info): commencing operation
krb5kdc: starting...
W20260929 03:10:16.990288 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:17 dist-test-slave-k1dg krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:17.122663 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:39879
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:39879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:17.238788 21715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:17.239074 21715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:17.239116 21715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:17.243404 21715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:17.243490 21715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:17.243510 21715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:17.243528 21715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:17.243543 21715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:17.249655 21715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:39879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:39879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:17.250938 21715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:17.252075 21715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:17.261708 21723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:17.261793 21715 server_base.cc:1034] running on GCE node
W20260929 03:10:17.261653 21720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:17.261703 21721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:17.262820 21715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:17.264099 21715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:17.265323 21715 hybrid_clock.cc:648] HybridClock initialized: now 1790651417265301 us; error 77 us; skew 500 ppm
Sep 29 03:10:17 dist-test-slave-k1dg krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651417, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:17.268762 21715 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:17.269901 21715 webserver.cc:460] Webserver started at http://127.20.174.126:40777/ using document root <none> and password file <none>
I20260929 03:10:17.270666 21715 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:17.270744 21715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:17.270994 21715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:17.273463 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "14a9a10e7bb949bb9db129b8ded00a82"
format_stamp: "Formatted at 2026-09-29 03:10:17 on dist-test-slave-k1dg"
server_key: "cc68d86f3b496519aeda769d11ede8db"
server_key_iv: "fd2133b5e4974d656e79e451b99ce3dc"
server_key_version: "encryptionkey@0"
I20260929 03:10:17.274158 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "14a9a10e7bb949bb9db129b8ded00a82"
format_stamp: "Formatted at 2026-09-29 03:10:17 on dist-test-slave-k1dg"
server_key: "cc68d86f3b496519aeda769d11ede8db"
server_key_iv: "fd2133b5e4974d656e79e451b99ce3dc"
server_key_version: "encryptionkey@0"
I20260929 03:10:17.278870 21715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 03:10:17.281888 21730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:17.283433 21715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:17.283569 21715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "14a9a10e7bb949bb9db129b8ded00a82"
format_stamp: "Formatted at 2026-09-29 03:10:17 on dist-test-slave-k1dg"
server_key: "cc68d86f3b496519aeda769d11ede8db"
server_key_iv: "fd2133b5e4974d656e79e451b99ce3dc"
server_key_version: "encryptionkey@0"
I20260929 03:10:17.283681 21715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:17.317724 21715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:17.321611 21715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:17.321789 21715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:17.329733 21715 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:39879
I20260929 03:10:17.329763 21782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:39879 every 8 connection(s)
I20260929 03:10:17.330910 21715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:17.334151 21783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:17.340296 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 21715
I20260929 03:10:17.340400 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:17.340704 21177 external_mini_cluster.cc:1428] Setting key e642f24511634f3384f05cb73bc7c2f1
I20260929 03:10:17.342567 21783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82: Bootstrap starting.
I20260929 03:10:17.345258 21783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:17.346226 21783 log.cc:824] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:17.348706 21783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82: No bootstrap required, opened a new log
Sep 29 03:10:17 dist-test-slave-k1dg krb5kdc[21702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:17.353314 21783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9a10e7bb949bb9db129b8ded00a82" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39879 } }
I20260929 03:10:17.353641 21783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:17.353716 21783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a9a10e7bb949bb9db129b8ded00a82, State: Initialized, Role: FOLLOWER
I20260929 03:10:17.354374 21783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9a10e7bb949bb9db129b8ded00a82" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39879 } }
I20260929 03:10:17.354579 21783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:17.354655 21783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:17.354769 21783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:17.356017 21783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9a10e7bb949bb9db129b8ded00a82" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39879 } }
I20260929 03:10:17.356405 21783 leader_election.cc:302] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14a9a10e7bb949bb9db129b8ded00a82; no voters:
I20260929 03:10:17.356781 21783 leader_election.cc:288] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:17.356920 21788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:17.357158 21788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [term 1 LEADER]: Becoming Leader. State: Replica: 14a9a10e7bb949bb9db129b8ded00a82, State: Running, Role: LEADER
I20260929 03:10:17.357581 21788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9a10e7bb949bb9db129b8ded00a82" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39879 } }
I20260929 03:10:17.358269 21783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:17.358987 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:17.342671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.20.174.126:39879)
0929 03:10:17.343248 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:17.343273 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:17.343318 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:10:17.344072 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:17.344091 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:17.344504 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:17.344952 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:17.345906 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:17.347426 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:17.348456 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:17.349068 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:17.349267 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:17.352572 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:17.352579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:17.352595 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:17.352629 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:17.354946 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:17.355458 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:17.355464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:17.355471 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:17.355623 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:17.355968 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:17.355972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:17.355974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:17.356535 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:17.356782 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:17.357553 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 03:10:17.357878 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":173,"threads_started":1}
I20260929 03:10:17.359404 21789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14a9a10e7bb949bb9db129b8ded00a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9a10e7bb949bb9db129b8ded00a82" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39879 } } }
I20260929 03:10:17.359541 21789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:17.359884 21790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14a9a10e7bb949bb9db129b8ded00a82. Latest consensus state: current_term: 1 leader_uuid: "14a9a10e7bb949bb9db129b8ded00a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9a10e7bb949bb9db129b8ded00a82" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39879 } } }
I20260929 03:10:17.360070 21790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82 [sys.catalog]: This master's current role is: LEADER
W20260929 03:10:17.363647 21803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:10:17.363727 21803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:10:17.363816 21797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:17.364532 21797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:17.371044 21797 catalog_manager.cc:1347] Generated new cluster ID: 286a255d6964484ab3499a71f6d9a579
I20260929 03:10:17.371145 21797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:17.385131 21797 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:17.386137 21797 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:17.391396 21797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14a9a10e7bb949bb9db129b8ded00a82: Generated new TSK 0
I20260929 03:10:17.392292 21797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:10:17.423736 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39879
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:17.542357 21809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:17.542670 21809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:17.542718 21809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:17.547067 21809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:17.547143 21809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:17.547207 21809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:17.552804 21809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39879
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:17.553906 21809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:17.555135 21809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:17.565323 21815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:17.565339 21814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:17.565390 21809 server_base.cc:1034] running on GCE node
W20260929 03:10:17.565351 21817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:17.566023 21809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:17.566874 21809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:17.568110 21809 hybrid_clock.cc:648] HybridClock initialized: now 1790651417568094 us; error 38 us; skew 500 ppm
Sep 29 03:10:17 dist-test-slave-k1dg krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651417, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:17.571244 21809 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:10:17.572118 21809 webserver.cc:460] Webserver started at http://127.20.174.65:44173/ using document root <none> and password file <none>
I20260929 03:10:17.572664 21809 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:17.572722 21809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:17.572891 21809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:17.575326 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "5fea0fa8cbc2433292b9004f7bc14739"
format_stamp: "Formatted at 2026-09-29 03:10:17 on dist-test-slave-k1dg"
server_key: "09317fbf98ad77e25d224e7dd4923896"
server_key_iv: "ccef6642810291ae67f82c2c106442ad"
server_key_version: "encryptionkey@0"
I20260929 03:10:17.575928 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "5fea0fa8cbc2433292b9004f7bc14739"
format_stamp: "Formatted at 2026-09-29 03:10:17 on dist-test-slave-k1dg"
server_key: "09317fbf98ad77e25d224e7dd4923896"
server_key_iv: "ccef6642810291ae67f82c2c106442ad"
server_key_version: "encryptionkey@0"
I20260929 03:10:17.579787 21809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:10:17.582638 21824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:17.583948 21809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:10:17.584141 21809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "5fea0fa8cbc2433292b9004f7bc14739"
format_stamp: "Formatted at 2026-09-29 03:10:17 on dist-test-slave-k1dg"
server_key: "09317fbf98ad77e25d224e7dd4923896"
server_key_iv: "ccef6642810291ae67f82c2c106442ad"
server_key_version: "encryptionkey@0"
I20260929 03:10:17.584254 21809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:17.596279 21809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:17.599766 21809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:17.599993 21809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:17.600677 21809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:17.601569 21809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:17.601619 21809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:17.601689 21809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:17.601722 21809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:17.611729 21809 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:33705
I20260929 03:10:17.611747 21937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:33705 every 8 connection(s)
I20260929 03:10:17.612774 21809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:17.621029 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 21809
I20260929 03:10:17.621124 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:17.621408 21177 external_mini_cluster.cc:1428] Setting key 231b5595b2875dc877086457feb812bc
Sep 29 03:10:17 dist-test-slave-k1dg krb5kdc[21702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651417, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:17.627691 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:17.614903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:37747 (local address 127.20.174.126:39879)
0929 03:10:17.615074 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:17.615078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:17.616006 (+ 928us) server_negotiation.cc:407] Connection header received
0929 03:10:17.616940 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:17.616943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:17.617002 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:17.617076 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:17.618294 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:17.618914 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:17.620106 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:17.620351 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:17.620484 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:17.623346 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:17.623350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:17.623353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:17.623385 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:17.625050 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:17.625790 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:17.625793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:17.625794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:17.625875 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:17.626291 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:17.626297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:17.626302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:17.626596 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:17.626703 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:17.627241 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 03:10:17.627402 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 03:10:17.628530 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:17.615334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39879 (local address 127.20.174.65:37747)
0929 03:10:17.615835 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:17.615879 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:17.616724 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:17.617264 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:17.617272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:17.617636 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:17.618137 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:17.618150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:17.619051 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:17.619055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:17.619960 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:17.619970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:17.620529 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:17.620533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:17.620704 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:17.621357 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:17.621374 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:17.623178 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:17.625179 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:17.625187 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:17.625207 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:17.625667 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:17.625975 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:17.625978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:17.625979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:17.626188 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:17.626701 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:17.626706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:17.626966 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 03:10:17.627155 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 03:10:17.627549 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":124,"threads_started":1}
I20260929 03:10:17.630052 21938 heartbeater.cc:347] Connected to a master server at 127.20.174.126:39879
I20260929 03:10:17.630347 21938 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:17.631042 21938 heartbeater.cc:511] Master 127.20.174.126:39879 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:10:17.633143 21747 ts_manager.cc:194] Registered new tserver with Master: 5fea0fa8cbc2433292b9004f7bc14739 (127.20.174.65:33705)
I20260929 03:10:17.634418 21747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:37747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:10:17.655511 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39879
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:17.781612 21943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:17.781896 21943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:17.781947 21943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:17.786355 21943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:17.786510 21943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:17.786621 21943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:17.792502 21943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39879
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:17.793687 21943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:17.794878 21943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:17.805361 21951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:17.805395 21943 server_base.cc:1034] running on GCE node
W20260929 03:10:17.805327 21948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:17.805503 21949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:17.806072 21943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:17.806813 21943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:17.808028 21943 hybrid_clock.cc:648] HybridClock initialized: now 1790651417808014 us; error 38 us; skew 500 ppm
Sep 29 03:10:17 dist-test-slave-k1dg krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651417, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:17.811204 21943 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:10:17.812150 21943 webserver.cc:460] Webserver started at http://127.20.174.66:35807/ using document root <none> and password file <none>
I20260929 03:10:17.812875 21943 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:17.812978 21943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:17.813210 21943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:17.815454 21943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "54d396febd1f408c864185e964ec8aef"
format_stamp: "Formatted at 2026-09-29 03:10:17 on dist-test-slave-k1dg"
server_key: "7e924e91b333f2a946753179754c8057"
server_key_iv: "4746be5bdff44ee13c4b1e849143d8f9"
server_key_version: "encryptionkey@0"
I20260929 03:10:17.816174 21943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "54d396febd1f408c864185e964ec8aef"
format_stamp: "Formatted at 2026-09-29 03:10:17 on dist-test-slave-k1dg"
server_key: "7e924e91b333f2a946753179754c8057"
server_key_iv: "4746be5bdff44ee13c4b1e849143d8f9"
server_key_version: "encryptionkey@0"
I20260929 03:10:17.820276 21943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 03:10:17.823009 21958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:17.824597 21943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:17.824747 21943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "54d396febd1f408c864185e964ec8aef"
format_stamp: "Formatted at 2026-09-29 03:10:17 on dist-test-slave-k1dg"
server_key: "7e924e91b333f2a946753179754c8057"
server_key_iv: "4746be5bdff44ee13c4b1e849143d8f9"
server_key_version: "encryptionkey@0"
I20260929 03:10:17.824877 21943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:17.843246 21943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:17.847708 21943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:17.848034 21943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:17.848793 21943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:17.849787 21943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:17.849874 21943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:17.849964 21943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:17.850015 21943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:17.860824 21943 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:43795
I20260929 03:10:17.860843 22071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:43795 every 8 connection(s)
I20260929 03:10:17.862107 21943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:17.863356 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 21943
I20260929 03:10:17.863449 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:17.863721 21177 external_mini_cluster.cc:1428] Setting key 54b864bb9919d8836c5f1b535f66aa7d
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
Sep 29 03:10:17 dist-test-slave-k1dg krb5kdc[21702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651417, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:10:17.878455 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:17.864253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:36713 (local address 127.20.174.126:39879)
0929 03:10:17.864418 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:17.864423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:17.865542 (+ 1119us) server_negotiation.cc:407] Connection header received
0929 03:10:17.866709 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:17.866714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:17.866782 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:17.866885 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:17.868585 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:17.869598 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:17.871103 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:17.871528 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:17.871664 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:17.874373 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:17.874376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:17.874378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:17.874406 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:17.876043 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:17.876562 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:17.876565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:17.876566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:17.876649 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:17.877222 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:17.877225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:17.877227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:17.877457 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:17.877579 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:17.878117 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 03:10:17.878248 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:10:17.879374 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:17.864733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39879 (local address 127.20.174.66:36713)
0929 03:10:17.865311 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:17.865397 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:17.866421 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:17.867114 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:17.867125 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:17.867690 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:17.868395 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:17.868412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:17.869794 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:17.869797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:17.870959 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:17.870971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:17.871704 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:17.871707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:17.871917 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:17.872586 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:17.872604 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:17.874175 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:17.876145 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:17.876150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:17.876165 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:17.876443 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:17.876900 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:17.876906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:17.876911 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:17.877114 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:17.877576 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:17.877583 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:17.877861 (+ 278us) client_negotiation.cc:769] Sending connection context
0929 03:10:17.878073 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 03:10:17.878470 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":170,"threads_started":1}
I20260929 03:10:17.880838 22072 heartbeater.cc:347] Connected to a master server at 127.20.174.126:39879
I20260929 03:10:17.881107 22072 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:17.881716 22072 heartbeater.cc:511] Master 127.20.174.126:39879 requested a full tablet report, sending...
I20260929 03:10:17.882997 21747 ts_manager.cc:194] Registered new tserver with Master: 54d396febd1f408c864185e964ec8aef (127.20.174.66:43795)
I20260929 03:10:17.883703 21747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:36713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:10:17.899344 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39879
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:18.020962 22077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:18.021273 22077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:18.021320 22077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:18.025738 22077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:18.025820 22077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:18.025892 22077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:18.032091 22077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39879
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:18.033241 22077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:18.034281 22077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:18.044878 22085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:18.044878 22082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:18.044895 22083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:18.044970 22077 server_base.cc:1034] running on GCE node
I20260929 03:10:18.045680 22077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:18.046545 22077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:18.047782 22077 hybrid_clock.cc:648] HybridClock initialized: now 1790651418047796 us; error 80 us; skew 500 ppm
Sep 29 03:10:18 dist-test-slave-k1dg krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651418, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:18.050954 22077 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:10:18.051846 22077 webserver.cc:460] Webserver started at http://127.20.174.67:32913/ using document root <none> and password file <none>
I20260929 03:10:18.052414 22077 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:18.052536 22077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:18.052809 22077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:18.055897 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "fff29210c987495f98eb2457fd415b92"
format_stamp: "Formatted at 2026-09-29 03:10:18 on dist-test-slave-k1dg"
server_key: "e76f150ef3b78ec67f6a5cfd0399dacf"
server_key_iv: "88a448f2fda348708554244851d1b6f8"
server_key_version: "encryptionkey@0"
I20260929 03:10:18.056447 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "fff29210c987495f98eb2457fd415b92"
format_stamp: "Formatted at 2026-09-29 03:10:18 on dist-test-slave-k1dg"
server_key: "e76f150ef3b78ec67f6a5cfd0399dacf"
server_key_iv: "88a448f2fda348708554244851d1b6f8"
server_key_version: "encryptionkey@0"
I20260929 03:10:18.061118 22077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:18.063886 22092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:18.065303 22077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:18.065410 22077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "fff29210c987495f98eb2457fd415b92"
format_stamp: "Formatted at 2026-09-29 03:10:18 on dist-test-slave-k1dg"
server_key: "e76f150ef3b78ec67f6a5cfd0399dacf"
server_key_iv: "88a448f2fda348708554244851d1b6f8"
server_key_version: "encryptionkey@0"
I20260929 03:10:18.065510 22077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:18.078255 22077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:18.081821 22077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:18.082050 22077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:18.082726 22077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:18.083513 22077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:18.083560 22077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:18.083599 22077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:18.083626 22077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:18.093489 22077 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:32783
I20260929 03:10:18.093518 22205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:32783 every 8 connection(s)
I20260929 03:10:18.094492 22077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:18.096565 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 22077
I20260929 03:10:18.096693 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:18.097003 21177 external_mini_cluster.cc:1428] Setting key cd453f24d99da4ec554076d729b3f0e5
Sep 29 03:10:18 dist-test-slave-k1dg krb5kdc[21702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651418, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:18.108575 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.096513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:39019 (local address 127.20.174.126:39879)
0929 03:10:18.096690 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:18.096696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:18.097778 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 03:10:18.098927 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:18.098931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:18.098988 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:18.099070 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:18.100270 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.100851 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.102077 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.102319 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.102439 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.104985 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:18.104989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:18.104992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:18.105020 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:18.106417 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:18.107046 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:18.107049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:18.107050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:18.107134 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:18.107493 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:18.107495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:18.107496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:18.107678 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:18.107772 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:18.108311 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 03:10:18.108416 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:10:18.109490 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.097001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39879 (local address 127.20.174.67:39019)
0929 03:10:18.097581 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:18.097654 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:18.098706 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:18.099248 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:18.099256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:18.099619 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:18.100108 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.100122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.101001 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.101007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.101922 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.101933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.102504 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.102509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.102683 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.103251 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:18.103268 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:18.104779 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:18.106649 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:18.106654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:18.106667 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:18.106952 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:18.107224 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:18.107226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:18.107227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:18.107380 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:18.107796 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:18.107801 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:18.108090 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 03:10:18.108256 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 03:10:18.108528 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":159,"threads_started":1}
I20260929 03:10:18.110759 22206 heartbeater.cc:347] Connected to a master server at 127.20.174.126:39879
I20260929 03:10:18.111039 22206 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:18.111575 22206 heartbeater.cc:511] Master 127.20.174.126:39879 requested a full tablet report, sending...
I20260929 03:10:18.112702 21746 ts_manager.cc:194] Registered new tserver with Master: fff29210c987495f98eb2457fd415b92 (127.20.174.67:32783)
I20260929 03:10:18.113332 21746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:39019
I20260929 03:10:18.123314 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:10:18.135346 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.126941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.20.174.126:39879)
0929 03:10:18.127089 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:18.127092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:18.127271 (+ 179us) server_negotiation.cc:407] Connection header received
0929 03:10:18.127367 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:18.127369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:18.127414 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:18.127514 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:18.128197 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.128835 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.129864 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.130096 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.130185 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.131856 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:18.131858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:18.131861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:18.131883 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:18.133430 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:18.133980 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:18.133983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:18.133985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:18.134084 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:18.134465 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:18.134469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:18.134471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:18.134707 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:18.134821 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:18.135118 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 03:10:18.135223 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:10:18.140954 21746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:18.144243 21746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:18.159729 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.153804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.20.174.66:43795)
0929 03:10:18.154244 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:18.154259 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:18.154679 (+ 420us) server_negotiation.cc:407] Connection header received
0929 03:10:18.154983 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:18.154989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:18.155216 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:18.155347 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:18.155940 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.156747 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.158150 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.158899 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.159022 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.159179 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:18.159417 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 03:10:18.159541 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":110,"threads_started":1}
I20260929 03:10:18.160300 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.153682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:43795 (local address 127.0.0.1:45332)
0929 03:10:18.154523 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:18.154584 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:18.154790 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:18.155580 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:18.155586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:18.155613 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:18.155812 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.155821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.156923 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.156925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.157984 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.158018 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.159019 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.159023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.159163 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.159203 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:10:18.159990 (+ 787us) client_negotiation.cc:240] Negotiation successful
0929 03:10:18.160133 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":65,"thread_start_us":209,"threads_started":1}
I20260929 03:10:18.161151 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.154399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:33705 (local address 127.0.0.1:47384)
0929 03:10:18.154875 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:18.154889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:18.154979 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:18.155598 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:18.155602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:18.155622 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:18.155827 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.155839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.157534 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.157538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.159262 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.159271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.160326 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.160329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.160848 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.160872 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:10:18.160947 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:10:18.161020 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":352,"thread_start_us":269,"threads_started":1}
I20260929 03:10:18.161311 22220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.154038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.20.174.67:32783)
0929 03:10:18.154518 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:18.154532 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:18.155154 (+ 622us) server_negotiation.cc:407] Connection header received
0929 03:10:18.155357 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:18.155362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:18.155550 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:18.155665 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:18.156143 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.157403 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.159244 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.160340 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.160428 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.160567 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:18.161054 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 03:10:18.161172 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":138,"threads_started":1}
I20260929 03:10:18.161954 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.154692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47384 (local address 127.20.174.65:33705)
0929 03:10:18.155104 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:18.155114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:18.155139 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:10:18.155223 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:18.155228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:18.155442 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:18.155612 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:18.155979 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.157375 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.159375 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.160181 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.160315 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.160496 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:18.161444 (+ 948us) server_negotiation.cc:299] Negotiation successful
0929 03:10:18.161648 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":92,"threads_started":1}
I20260929 03:10:18.161767 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.153668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:32783 (local address 127.0.0.1:33920)
0929 03:10:18.155084 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:18.155095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:18.155213 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:18.155850 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:18.155855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:18.155876 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:18.155976 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.155984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.157548 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.157551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.159064 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.159075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.160787 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.160791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.160883 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.160904 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:10:18.161577 (+ 673us) client_negotiation.cc:240] Negotiation successful
0929 03:10:18.161654 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":9,"thread_start_us":81,"threads_started":1}
I20260929 03:10:18.163802 22006 tablet_service.cc:1467] Processing CreateTablet for tablet ee6f2a968243498380e5e4eb8a7bc94d (DEFAULT_TABLE table=test-table [id=e4d9ccee5f4e407e9b0f8476d281ba40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:18.165096 22006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6f2a968243498380e5e4eb8a7bc94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:18.164935 21872 tablet_service.cc:1467] Processing CreateTablet for tablet ee6f2a968243498380e5e4eb8a7bc94d (DEFAULT_TABLE table=test-table [id=e4d9ccee5f4e407e9b0f8476d281ba40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:18.164935 22140 tablet_service.cc:1467] Processing CreateTablet for tablet ee6f2a968243498380e5e4eb8a7bc94d (DEFAULT_TABLE table=test-table [id=e4d9ccee5f4e407e9b0f8476d281ba40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:18.165997 21872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6f2a968243498380e5e4eb8a7bc94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:18.165998 22140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6f2a968243498380e5e4eb8a7bc94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:18.171900 22223 tablet_bootstrap.cc:492] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef: Bootstrap starting.
I20260929 03:10:18.173141 22224 tablet_bootstrap.cc:492] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92: Bootstrap starting.
I20260929 03:10:18.173978 22225 tablet_bootstrap.cc:492] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739: Bootstrap starting.
I20260929 03:10:18.174324 22223 tablet_bootstrap.cc:654] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:18.175489 22224 tablet_bootstrap.cc:654] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:18.175704 22223 log.cc:824] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:18.176031 22225 tablet_bootstrap.cc:654] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:18.176692 22224 log.cc:824] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:18.176846 22225 log.cc:824] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:18.178540 22223 tablet_bootstrap.cc:492] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef: No bootstrap required, opened a new log
I20260929 03:10:18.178764 22223 ts_tablet_manager.cc:1397] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 03:10:18.178830 22224 tablet_bootstrap.cc:492] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92: No bootstrap required, opened a new log
I20260929 03:10:18.178830 22225 tablet_bootstrap.cc:492] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739: No bootstrap required, opened a new log
I20260929 03:10:18.179023 22224 ts_tablet_manager.cc:1397] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:10:18.179023 22225 ts_tablet_manager.cc:1397] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260929 03:10:18.182147 22223 raft_consensus.cc:348] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d396febd1f408c864185e964ec8aef" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43795 } } peers { permanent_uuid: "fff29210c987495f98eb2457fd415b92" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32783 } } peers { permanent_uuid: "5fea0fa8cbc2433292b9004f7bc14739" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33705 } }
I20260929 03:10:18.182648 22223 raft_consensus.cc:374] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:18.182739 22223 raft_consensus.cc:729] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d396febd1f408c864185e964ec8aef, State: Initialized, Role: FOLLOWER
I20260929 03:10:18.183393 22223 consensus_queue.cc:260] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d396febd1f408c864185e964ec8aef" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43795 } } peers { permanent_uuid: "fff29210c987495f98eb2457fd415b92" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32783 } } peers { permanent_uuid: "5fea0fa8cbc2433292b9004f7bc14739" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33705 } }
I20260929 03:10:18.183264 22224 raft_consensus.cc:348] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d396febd1f408c864185e964ec8aef" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43795 } } peers { permanent_uuid: "fff29210c987495f98eb2457fd415b92" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32783 } } peers { permanent_uuid: "5fea0fa8cbc2433292b9004f7bc14739" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33705 } }
I20260929 03:10:18.183585 22224 raft_consensus.cc:374] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:18.183660 22224 raft_consensus.cc:729] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff29210c987495f98eb2457fd415b92, State: Initialized, Role: FOLLOWER
I20260929 03:10:18.184165 22072 heartbeater.cc:503] Master 127.20.174.126:39879 was elected leader, sending a full tablet report...
I20260929 03:10:18.184120 22224 consensus_queue.cc:260] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d396febd1f408c864185e964ec8aef" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43795 } } peers { permanent_uuid: "fff29210c987495f98eb2457fd415b92" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32783 } } peers { permanent_uuid: "5fea0fa8cbc2433292b9004f7bc14739" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33705 } }
I20260929 03:10:18.184562 22223 ts_tablet_manager.cc:1428] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:10:18.184460 22225 raft_consensus.cc:348] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d396febd1f408c864185e964ec8aef" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43795 } } peers { permanent_uuid: "fff29210c987495f98eb2457fd415b92" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32783 } } peers { permanent_uuid: "5fea0fa8cbc2433292b9004f7bc14739" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33705 } }
I20260929 03:10:18.184705 22225 raft_consensus.cc:374] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:18.184752 22225 raft_consensus.cc:729] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fea0fa8cbc2433292b9004f7bc14739, State: Initialized, Role: FOLLOWER
I20260929 03:10:18.185076 22224 ts_tablet_manager.cc:1428] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:10:18.185184 22225 consensus_queue.cc:260] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d396febd1f408c864185e964ec8aef" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43795 } } peers { permanent_uuid: "fff29210c987495f98eb2457fd415b92" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32783 } } peers { permanent_uuid: "5fea0fa8cbc2433292b9004f7bc14739" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33705 } }
I20260929 03:10:18.185592 22206 heartbeater.cc:503] Master 127.20.174.126:39879 was elected leader, sending a full tablet report...
I20260929 03:10:18.186074 22225 ts_tablet_manager.cc:1428] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 03:10:18.186407 21938 heartbeater.cc:503] Master 127.20.174.126:39879 was elected leader, sending a full tablet report...
I20260929 03:10:18.291060 22229 raft_consensus.cc:482] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:18.291307 22229 raft_consensus.cc:504] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d396febd1f408c864185e964ec8aef" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43795 } } peers { permanent_uuid: "fff29210c987495f98eb2457fd415b92" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32783 } } peers { permanent_uuid: "5fea0fa8cbc2433292b9004f7bc14739" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33705 } }
I20260929 03:10:18.292594 22229 leader_election.cc:288] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fff29210c987495f98eb2457fd415b92 (127.20.174.67:32783), 5fea0fa8cbc2433292b9004f7bc14739 (127.20.174.65:33705)
I20260929 03:10:18.296357 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.292895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:32783 (local address 127.20.174.66:60177)
0929 03:10:18.293046 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:18.293062 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:18.293164 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:18.293458 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:18.293461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:18.293479 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:18.293616 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.293622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.294353 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.294358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.295242 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.295251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.295918 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.295922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.296032 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.296054 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:10:18.296152 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 03:10:18.296227 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 03:10:18.296357 22220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.293045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:60177 (local address 127.20.174.67:32783)
0929 03:10:18.293186 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:18.293191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:18.293211 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:10:18.293274 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:18.293278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:18.293330 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:18.293465 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:18.293737 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.294239 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.295376 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.295790 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.295925 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.295979 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:18.296168 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 03:10:18.296235 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:10:18.297034 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.293200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:33705 (local address 127.20.174.66:34279)
0929 03:10:18.293672 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:18.293682 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:18.293815 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:18.294083 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:18.294085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:18.294102 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:18.294234 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.294240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.294984 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.294989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.295764 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:18.295782 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.296609 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.296613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:18.296737 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.296761 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:10:18.296854 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:10:18.296930 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":231,"threads_started":1}
I20260929 03:10:18.297151 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.293298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:34279 (local address 127.20.174.65:33705)
0929 03:10:18.293492 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:18.293496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:18.293781 (+ 285us) server_negotiation.cc:407] Connection header received
0929 03:10:18.293936 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:18.293939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:18.293987 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:18.294070 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:18.294352 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.294864 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.295910 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.296341 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.296492 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.296561 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:18.296935 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 03:10:18.297027 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 03:10:18.297240 22160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6f2a968243498380e5e4eb8a7bc94d" candidate_uuid: "54d396febd1f408c864185e964ec8aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff29210c987495f98eb2457fd415b92" is_pre_election: true
I20260929 03:10:18.297617 22160 raft_consensus.cc:2454] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54d396febd1f408c864185e964ec8aef in term 0.
I20260929 03:10:18.297801 21892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6f2a968243498380e5e4eb8a7bc94d" candidate_uuid: "54d396febd1f408c864185e964ec8aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fea0fa8cbc2433292b9004f7bc14739" is_pre_election: true
I20260929 03:10:18.298108 21892 raft_consensus.cc:2454] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54d396febd1f408c864185e964ec8aef in term 0.
I20260929 03:10:18.298334 21962 leader_election.cc:302] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54d396febd1f408c864185e964ec8aef, fff29210c987495f98eb2457fd415b92; no voters:
I20260929 03:10:18.298681 22229 raft_consensus.cc:2781] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:18.298785 22229 raft_consensus.cc:482] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:18.298847 22229 raft_consensus.cc:3037] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:18.299947 22229 raft_consensus.cc:504] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d396febd1f408c864185e964ec8aef" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43795 } } peers { permanent_uuid: "fff29210c987495f98eb2457fd415b92" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32783 } } peers { permanent_uuid: "5fea0fa8cbc2433292b9004f7bc14739" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33705 } }
I20260929 03:10:18.300313 22229 leader_election.cc:288] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [CANDIDATE]: Term 1 election: Requested vote from peers fff29210c987495f98eb2457fd415b92 (127.20.174.67:32783), 5fea0fa8cbc2433292b9004f7bc14739 (127.20.174.65:33705)
I20260929 03:10:18.300734 22160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6f2a968243498380e5e4eb8a7bc94d" candidate_uuid: "54d396febd1f408c864185e964ec8aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff29210c987495f98eb2457fd415b92"
I20260929 03:10:18.300881 22160 raft_consensus.cc:3037] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:18.300956 21892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6f2a968243498380e5e4eb8a7bc94d" candidate_uuid: "54d396febd1f408c864185e964ec8aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fea0fa8cbc2433292b9004f7bc14739"
I20260929 03:10:18.301128 21892 raft_consensus.cc:3037] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:18.302030 22160 raft_consensus.cc:2454] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54d396febd1f408c864185e964ec8aef in term 1.
I20260929 03:10:18.302404 21962 leader_election.cc:302] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54d396febd1f408c864185e964ec8aef, fff29210c987495f98eb2457fd415b92; no voters:
I20260929 03:10:18.302467 21892 raft_consensus.cc:2454] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54d396febd1f408c864185e964ec8aef in term 1.
I20260929 03:10:18.302685 22229 raft_consensus.cc:2781] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:18.302943 22229 raft_consensus.cc:686] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [term 1 LEADER]: Becoming Leader. State: Replica: 54d396febd1f408c864185e964ec8aef, State: Running, Role: LEADER
I20260929 03:10:18.303363 22229 consensus_queue.cc:237] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d396febd1f408c864185e964ec8aef" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43795 } } peers { permanent_uuid: "fff29210c987495f98eb2457fd415b92" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32783 } } peers { permanent_uuid: "5fea0fa8cbc2433292b9004f7bc14739" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33705 } }
I20260929 03:10:18.307386 21745 catalog_manager.cc:5495] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef reported cstate change: term changed from 0 to 1, leader changed from <none> to 54d396febd1f408c864185e964ec8aef (127.20.174.66). New cstate: current_term: 1 leader_uuid: "54d396febd1f408c864185e964ec8aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54d396febd1f408c864185e964ec8aef" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fff29210c987495f98eb2457fd415b92" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fea0fa8cbc2433292b9004f7bc14739" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33705 } health_report { overall_health: UNKNOWN } } }
W20260929 03:10:18.345551 22207 tablet.cc:2369] T ee6f2a968243498380e5e4eb8a7bc94d P fff29210c987495f98eb2457fd415b92: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 03:10:18 dist-test-slave-k1dg krb5kdc[21702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:10:18.354794 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.344792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.20.174.65:33705)
0929 03:10:18.345025 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:18.345029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:18.345153 (+ 124us) server_negotiation.cc:407] Connection header received
0929 03:10:18.345274 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:18.345277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:18.345346 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:18.345447 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:18.346223 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.346816 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.347702 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.347927 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.348019 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.350841 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:18.350853 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:18.350869 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:18.350898 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:18.352860 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:18.353332 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:18.353338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:18.353339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:18.353422 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:18.353749 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:18.353752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:18.353753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:18.354080 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:18.354279 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:18.354558 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:10:18.354667 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
W20260929 03:10:18.363412 22073 tablet.cc:2369] T ee6f2a968243498380e5e4eb8a7bc94d P 54d396febd1f408c864185e964ec8aef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:10:18.364470 21939 tablet.cc:2369] T ee6f2a968243498380e5e4eb8a7bc94d P 5fea0fa8cbc2433292b9004f7bc14739: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 03:10:18 dist-test-slave-k1dg krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651418, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:10:18 dist-test-slave-k1dg krb5kdc[21702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651418, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:10:18.384312 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.374272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.20.174.65:33705)
0929 03:10:18.374510 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:18.374515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:18.374632 (+ 117us) server_negotiation.cc:407] Connection header received
0929 03:10:18.374821 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:18.374829 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:18.374879 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:18.374970 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:18.375800 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.376341 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.377204 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.377458 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.377572 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.380213 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:18.380217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:18.380219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:18.380256 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:18.382296 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:18.382850 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:18.382854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:18.382855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:18.382980 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:18.383346 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:18.383349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:18.383351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:18.383543 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:18.383655 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:18.384051 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 03:10:18.384174 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
Sep 29 03:10:18 dist-test-slave-k1dg krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:10:18 dist-test-slave-k1dg krb5kdc[21702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:10:18.411635 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:18.402901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.20.174.65:33705)
0929 03:10:18.403058 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:18.403064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:18.403252 (+ 188us) server_negotiation.cc:407] Connection header received
0929 03:10:18.403458 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:18.403462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:18.403508 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:18.403607 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:18.404362 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.404868 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.405732 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:18.405964 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:18.406103 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:18.408689 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:18.408692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:18.408694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:18.408722 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:18.410197 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:18.410674 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:18.410677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:18.410679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:18.410757 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:18.410981 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:18.410983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:18.410984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:18.411136 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:18.411229 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:18.411406 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:10:18.411509 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 03:10:18.413161 21852 tablet_service.cc:2908] Rejecting scan request for tablet ee6f2a968243498380e5e4eb8a7bc94d: Uninitialized: clean time has not yet been initialized
I20260929 03:10:18.415536 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 21809
I20260929 03:10:18.421515 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 21943
I20260929 03:10:18.427456 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 22077
I20260929 03:10:18.432961 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 21715
2026-09-29T03:10:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2573 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:18 dist-test-slave-k1dg krb5kdc[22247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:18 dist-test-slave-k1dg krb5kdc[22247](info): set up 2 sockets
Sep 29 03:10:18 dist-test-slave-k1dg krb5kdc[22247](info): commencing operation
krb5kdc: starting...
W20260929 03:10:19.538367 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:19 dist-test-slave-k1dg krb5kdc[22247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:19.669883 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:41111
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:41111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:19.792454 22260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:19.792739 22260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:19.792780 22260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:19.797004 22260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:19.797072 22260 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:10:19.797091 22260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:19.797108 22260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:19.797125 22260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:19.797174 22260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:19.803537 22260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:41111
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:19.804800 22260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:19.805970 22260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:19.815671 22266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:19.815646 22268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:19.815864 22260 server_base.cc:1034] running on GCE node
W20260929 03:10:19.815640 22265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:19.816735 22260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:19.817991 22260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:19.819244 22260 hybrid_clock.cc:648] HybridClock initialized: now 1790651419819221 us; error 34 us; skew 500 ppm
Sep 29 03:10:19 dist-test-slave-k1dg krb5kdc[22247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651419, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:19.830879 22260 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:19.832046 22260 webserver.cc:460] Webserver started at http://127.20.174.126:45329/ using document root <none> and password file <none>
I20260929 03:10:19.832724 22260 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:19.832844 22260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:19.833103 22260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:19.835500 22260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "89d9e5c474ec4b43bc37e3bf39d20d91"
format_stamp: "Formatted at 2026-09-29 03:10:19 on dist-test-slave-k1dg"
server_key: "f195a37d1f710e839ced96e1f094002c"
server_key_iv: "027af1232672a64355f1dc4432c4397d"
server_key_version: "encryptionkey@0"
I20260929 03:10:19.836151 22260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "89d9e5c474ec4b43bc37e3bf39d20d91"
format_stamp: "Formatted at 2026-09-29 03:10:19 on dist-test-slave-k1dg"
server_key: "f195a37d1f710e839ced96e1f094002c"
server_key_iv: "027af1232672a64355f1dc4432c4397d"
server_key_version: "encryptionkey@0"
I20260929 03:10:19.840588 22260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:19.843627 22275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:19.845274 22260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:10:19.845464 22260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "89d9e5c474ec4b43bc37e3bf39d20d91"
format_stamp: "Formatted at 2026-09-29 03:10:19 on dist-test-slave-k1dg"
server_key: "f195a37d1f710e839ced96e1f094002c"
server_key_iv: "027af1232672a64355f1dc4432c4397d"
server_key_version: "encryptionkey@0"
I20260929 03:10:19.845619 22260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:19.865496 22260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:19.869089 22260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:19.869314 22260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:19.877846 22260 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:41111
I20260929 03:10:19.877862 22337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:41111 every 8 connection(s)
I20260929 03:10:19.878834 22260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:19.881783 22338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:19.887534 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 22260
I20260929 03:10:19.887662 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:19.887876 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91: Bootstrap starting.
I20260929 03:10:19.887951 21177 external_mini_cluster.cc:1428] Setting key dbbf8957355b24a9b6c7bccbdabe2a06
I20260929 03:10:19.890620 22338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:19.891409 22338 log.cc:824] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:19.893443 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91: No bootstrap required, opened a new log
Sep 29 03:10:19 dist-test-slave-k1dg krb5kdc[22247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:19.897564 22338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9e5c474ec4b43bc37e3bf39d20d91" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41111 } }
I20260929 03:10:19.897877 22338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:19.898008 22338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d9e5c474ec4b43bc37e3bf39d20d91, State: Initialized, Role: FOLLOWER
I20260929 03:10:19.898594 22338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9e5c474ec4b43bc37e3bf39d20d91" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41111 } }
I20260929 03:10:19.898782 22338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:19.898852 22338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:19.898985 22338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:19.900291 22338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9e5c474ec4b43bc37e3bf39d20d91" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41111 } }
I20260929 03:10:19.900688 22338 leader_election.cc:302] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89d9e5c474ec4b43bc37e3bf39d20d91; no voters:
I20260929 03:10:19.901034 22338 leader_election.cc:288] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:19.901153 22343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:19.901393 22343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [term 1 LEADER]: Becoming Leader. State: Replica: 89d9e5c474ec4b43bc37e3bf39d20d91, State: Running, Role: LEADER
I20260929 03:10:19.901748 22343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9e5c474ec4b43bc37e3bf39d20d91" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41111 } }
I20260929 03:10:19.902506 22338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:19.903702 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89d9e5c474ec4b43bc37e3bf39d20d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9e5c474ec4b43bc37e3bf39d20d91" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41111 } } }
I20260929 03:10:19.903854 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:19.903702 22345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89d9e5c474ec4b43bc37e3bf39d20d91. Latest consensus state: current_term: 1 leader_uuid: "89d9e5c474ec4b43bc37e3bf39d20d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d9e5c474ec4b43bc37e3bf39d20d91" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41111 } } }
I20260929 03:10:19.904214 22345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:19.905325 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:19.889509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.20.174.126:41111)
0929 03:10:19.889986 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:19.890001 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:19.890040 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:10:19.890843 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:19.890864 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:19.891232 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:19.891640 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:19.892483 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:19.893489 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:19.894470 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:19.895311 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:19.895512 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:19.898339 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:19.898346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:19.898363 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:19.898396 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:19.900635 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:19.901215 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:19.901223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:19.901230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:19.901379 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:19.901733 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:19.901736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:19.901737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:19.902278 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:19.902503 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:19.904026 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0929 03:10:19.904365 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":139,"threads_started":1}
W20260929 03:10:19.907516 22358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:10:19.907627 22358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:10:19.908166 22352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:19.908946 22352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:19.915107 22352 catalog_manager.cc:1347] Generated new cluster ID: 16c15ccb82a14bfdbcbf11e31f91d493
I20260929 03:10:19.915198 22352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:19.935248 22352 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:19.936618 22352 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:19.952544 22352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91: Generated new TSK 0
I20260929 03:10:19.953490 22352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:10:19.986899 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:41111
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:20.104923 22364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:20.105238 22364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:20.105314 22364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:20.109673 22364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:20.109799 22364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:20.109871 22364 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:10:20.109959 22364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:20.115506 22364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.174.126:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:20.116699 22364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:20.117830 22364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:20.128563 22370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:20.128563 22372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:20.128624 22364 server_base.cc:1034] running on GCE node
W20260929 03:10:20.128567 22369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:20.129276 22364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:20.130026 22364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:20.131227 22364 hybrid_clock.cc:648] HybridClock initialized: now 1790651420131177 us; error 66 us; skew 500 ppm
Sep 29 03:10:20 dist-test-slave-k1dg krb5kdc[22247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651420, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:20.134225 22364 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:10:20.135316 22364 webserver.cc:460] Webserver started at http://127.20.174.65:45151/ using document root <none> and password file <none>
I20260929 03:10:20.135939 22364 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:20.136044 22364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:20.136293 22364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:20.138594 22364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "2815b56ae26f48cb91a29df1f0794f74"
format_stamp: "Formatted at 2026-09-29 03:10:20 on dist-test-slave-k1dg"
server_key: "392843b389a70f2bffcf75c6a321e00b"
server_key_iv: "ec924372a73ce6842ffb5edd16709005"
server_key_version: "encryptionkey@0"
I20260929 03:10:20.139213 22364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "2815b56ae26f48cb91a29df1f0794f74"
format_stamp: "Formatted at 2026-09-29 03:10:20 on dist-test-slave-k1dg"
server_key: "392843b389a70f2bffcf75c6a321e00b"
server_key_iv: "ec924372a73ce6842ffb5edd16709005"
server_key_version: "encryptionkey@0"
I20260929 03:10:20.143503 22364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:10:20.146323 22379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:20.148056 22364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:10:20.148175 22364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "2815b56ae26f48cb91a29df1f0794f74"
format_stamp: "Formatted at 2026-09-29 03:10:20 on dist-test-slave-k1dg"
server_key: "392843b389a70f2bffcf75c6a321e00b"
server_key_iv: "ec924372a73ce6842ffb5edd16709005"
server_key_version: "encryptionkey@0"
I20260929 03:10:20.148352 22364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:20.161332 22364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:20.165141 22364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:20.165369 22364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:20.167057 22364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:20.167110 22364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:20.167181 22364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:20.167213 22364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:10:20 dist-test-slave-k1dg krb5kdc[22247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651420, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:20.178655 22364 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:40431
I20260929 03:10:20.178721 22494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:40431 every 8 connection(s)
I20260929 03:10:20.179899 22364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:20.182927 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.168083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:35827 (local address 127.20.174.126:41111)
0929 03:10:20.168253 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.168258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.169144 (+ 886us) server_negotiation.cc:407] Connection header received
0929 03:10:20.170313 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.170317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.170383 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.170524 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:20.172311 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.173198 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.174811 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.175074 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.175203 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.178382 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:20.178387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:20.178390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:20.178426 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:20.180147 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.180837 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.180841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.180842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.180926 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.181497 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.181500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.181501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.181703 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:20.181810 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.182385 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.182795 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:10:20.183624 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.168483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:41111 (local address 127.20.174.65:35827)
0929 03:10:20.168954 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.169020 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.170058 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.170741 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.170754 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.171278 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:20.172079 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.172108 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.173367 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.173371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.174672 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.174685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.175251 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.175255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.175449 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.176182 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:20.176206 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:20.178157 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:20.180278 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.180285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.180303 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.180693 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.181131 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.181133 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.181135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.181310 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.181860 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:20.181866 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:20.182124 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.182299 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.182678 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":139,"threads_started":1}
I20260929 03:10:20.184651 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 22364
I20260929 03:10:20.184793 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:20.185056 21177 external_mini_cluster.cc:1428] Setting key 13026999a38d2501d5e55fec890bca21
I20260929 03:10:20.185230 22495 heartbeater.cc:347] Connected to a master server at 127.20.174.126:41111
I20260929 03:10:20.185381 22495 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:20.185957 22495 heartbeater.cc:511] Master 127.20.174.126:41111 requested a full tablet report, sending...
I20260929 03:10:20.187896 22291 ts_manager.cc:194] Registered new tserver with Master: 2815b56ae26f48cb91a29df1f0794f74 (127.20.174.65:40431)
I20260929 03:10:20.189172 22291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:35827
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:10:20.196542 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.187678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:38353 (local address 127.20.174.126:41111)
0929 03:10:20.187814 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.187819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.188061 (+ 242us) server_negotiation.cc:407] Connection header received
0929 03:10:20.188248 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.188251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.188295 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.188386 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:20.188855 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.189710 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.191146 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.191473 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.191579 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.193015 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:20.193018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:20.193020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:20.193043 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:20.194687 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.195222 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.195224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.195226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.195298 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.195619 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.195622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.195623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.195794 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:20.195867 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.196310 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.196423 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:10:20.196574 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.187621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:41111 (local address 127.20.174.65:38353)
0929 03:10:20.187962 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.187975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.188131 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.188402 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.188405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.188625 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:20.188733 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.188739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.189883 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.189889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.190999 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.191015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.191629 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.191636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.191761 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.192315 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:20.192332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:20.192856 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:20.194802 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.194808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.194810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.195102 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.195398 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.195402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.195406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.195513 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.195891 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:20.195894 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:20.196010 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.196141 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.196316 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:10:20.219630 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:41111
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:20.336275 22504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:20.336565 22504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:20.336612 22504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:20.341039 22504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:20.341128 22504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:20.341171 22504 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:10:20.341207 22504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:20.347654 22504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.174.126:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:20.348914 22504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:20.350597 22504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:20.362160 22509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:20.362162 22512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:20.362331 22504 server_base.cc:1034] running on GCE node
W20260929 03:10:20.362588 22510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:20.363121 22504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:20.363853 22504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:20.365125 22504 hybrid_clock.cc:648] HybridClock initialized: now 1790651420365079 us; error 59 us; skew 500 ppm
Sep 29 03:10:20 dist-test-slave-k1dg krb5kdc[22247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651420, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:20.368355 22504 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:10:20.369347 22504 webserver.cc:460] Webserver started at http://127.20.174.66:39575/ using document root <none> and password file <none>
I20260929 03:10:20.369930 22504 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:20.369997 22504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:20.370174 22504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:20.372423 22504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "39897749def34c9f97fbcdf8570d9c04"
format_stamp: "Formatted at 2026-09-29 03:10:20 on dist-test-slave-k1dg"
server_key: "69eb748f2299987eba9a60323570a14e"
server_key_iv: "fc7e3134dd3ca90ddb3b1eb5aabb57f1"
server_key_version: "encryptionkey@0"
I20260929 03:10:20.372991 22504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "39897749def34c9f97fbcdf8570d9c04"
format_stamp: "Formatted at 2026-09-29 03:10:20 on dist-test-slave-k1dg"
server_key: "69eb748f2299987eba9a60323570a14e"
server_key_iv: "fc7e3134dd3ca90ddb3b1eb5aabb57f1"
server_key_version: "encryptionkey@0"
I20260929 03:10:20.377619 22504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 03:10:20.380525 22519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:20.382027 22504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:20.382136 22504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "39897749def34c9f97fbcdf8570d9c04"
format_stamp: "Formatted at 2026-09-29 03:10:20 on dist-test-slave-k1dg"
server_key: "69eb748f2299987eba9a60323570a14e"
server_key_iv: "fc7e3134dd3ca90ddb3b1eb5aabb57f1"
server_key_version: "encryptionkey@0"
I20260929 03:10:20.382234 22504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:20.393952 22504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:20.397506 22504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:20.397675 22504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:20.399302 22504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:20.399362 22504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:20.399407 22504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:20.399449 22504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:10:20 dist-test-slave-k1dg krb5kdc[22247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651420, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:20.411644 22504 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:34615
I20260929 03:10:20.411700 22634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:34615 every 8 connection(s)
I20260929 03:10:20.413103 22504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:20.415686 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.400261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:39059 (local address 127.20.174.126:41111)
0929 03:10:20.400446 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.400450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.401572 (+ 1122us) server_negotiation.cc:407] Connection header received
0929 03:10:20.403044 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.403047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.403112 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.403227 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:20.404902 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.405858 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.407396 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.407753 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.407899 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.411334 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:20.411340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:20.411345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:20.411385 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:20.413286 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.414038 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.414043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.414045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.414151 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.414594 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.414597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.414599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.414795 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:20.414877 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.415426 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.415558 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 03:10:20.416595 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.400844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:41111 (local address 127.20.174.66:39059)
0929 03:10:20.401390 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.401438 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.402798 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.403417 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.403427 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.403985 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:20.404618 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.404633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.406035 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.406040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.407228 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.407240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.407960 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.407964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.408147 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.409372 (+ 1225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:20.409398 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:20.411075 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:20.413415 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.413420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.413433 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.413908 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.414278 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.414281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.414282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.414497 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.414889 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:20.414894 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:20.415138 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.415293 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.415569 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":145,"threads_started":1}
I20260929 03:10:20.417102 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 22504
I20260929 03:10:20.417366 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:20.417716 21177 external_mini_cluster.cc:1428] Setting key 43c15ea508b3b25490b04a181f5a8b64
I20260929 03:10:20.420547 22635 heartbeater.cc:347] Connected to a master server at 127.20.174.126:41111
I20260929 03:10:20.420765 22635 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:20.421306 22635 heartbeater.cc:511] Master 127.20.174.126:41111 requested a full tablet report, sending...
I20260929 03:10:20.423202 22292 ts_manager.cc:194] Registered new tserver with Master: 39897749def34c9f97fbcdf8570d9c04 (127.20.174.66:34615)
I20260929 03:10:20.424099 22292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:39059
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:10:20.429656 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.420929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:53889 (local address 127.20.174.126:41111)
0929 03:10:20.421076 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.421082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.421386 (+ 304us) server_negotiation.cc:407] Connection header received
0929 03:10:20.421511 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.421514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.421562 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.421649 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:20.422266 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.423133 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.424094 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.424395 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.424513 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.425840 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:20.425846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:20.425852 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:20.425885 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:20.427828 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.428318 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.428321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.428323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.428419 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.428736 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.428741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.428743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.428984 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:20.429105 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.429368 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.429523 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:10:20.429656 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.420766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:41111 (local address 127.20.174.66:53889)
0929 03:10:20.421263 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.421281 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.421402 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.421694 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.421699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.421963 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:20.422118 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.422126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.423278 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.423283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.423969 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.423978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.424515 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.424519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.424613 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.425226 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:20.425242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:20.425664 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:20.427987 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.427989 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.427991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.428206 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.428517 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.428519 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.428520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.428600 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.429092 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:20.429095 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:20.429191 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.429303 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.429455 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:10:20.452406 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:41111
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:20.573251 22644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:20.573549 22644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:20.573596 22644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:20.577986 22644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:20.578096 22644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:20.578140 22644 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:10:20.578249 22644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:20.583889 22644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.174.126:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:20.585099 22644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:20.586352 22644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:20.597326 22650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:20.597422 22649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:20.597452 22644 server_base.cc:1034] running on GCE node
W20260929 03:10:20.597350 22652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:20.598187 22644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:20.598946 22644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:20.600127 22644 hybrid_clock.cc:648] HybridClock initialized: now 1790651420600090 us; error 51 us; skew 500 ppm
Sep 29 03:10:20 dist-test-slave-k1dg krb5kdc[22247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651420, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:20.603437 22644 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:10:20.604496 22644 webserver.cc:460] Webserver started at http://127.20.174.67:41437/ using document root <none> and password file <none>
I20260929 03:10:20.605108 22644 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:20.605175 22644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:20.605406 22644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:20.607748 22644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "21f5c489f32346668e7cb987eeecd5de"
format_stamp: "Formatted at 2026-09-29 03:10:20 on dist-test-slave-k1dg"
server_key: "ab02575316ee95ee7ad48c536d731669"
server_key_iv: "9744884e22e9f07e62c46147523eae6c"
server_key_version: "encryptionkey@0"
I20260929 03:10:20.608371 22644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "21f5c489f32346668e7cb987eeecd5de"
format_stamp: "Formatted at 2026-09-29 03:10:20 on dist-test-slave-k1dg"
server_key: "ab02575316ee95ee7ad48c536d731669"
server_key_iv: "9744884e22e9f07e62c46147523eae6c"
server_key_version: "encryptionkey@0"
I20260929 03:10:20.612702 22644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:20.615523 22659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:20.616878 22644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 03:10:20.617110 22644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "21f5c489f32346668e7cb987eeecd5de"
format_stamp: "Formatted at 2026-09-29 03:10:20 on dist-test-slave-k1dg"
server_key: "ab02575316ee95ee7ad48c536d731669"
server_key_iv: "9744884e22e9f07e62c46147523eae6c"
server_key_version: "encryptionkey@0"
I20260929 03:10:20.617249 22644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:20.629208 22644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:20.632687 22644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:20.632925 22644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:20.634543 22644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:20.634619 22644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:20.634681 22644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:20.634733 22644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:10:20 dist-test-slave-k1dg krb5kdc[22247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651420, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:20.646341 22644 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:36881
I20260929 03:10:20.646358 22774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:36881 every 8 connection(s)
I20260929 03:10:20.647403 22644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:20.649210 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.635515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:46253 (local address 127.20.174.126:41111)
0929 03:10:20.635726 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.635732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.636593 (+ 861us) server_negotiation.cc:407] Connection header received
0929 03:10:20.637646 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.637652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.637707 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.637799 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:20.639314 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.639968 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.641240 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.641629 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.641756 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.645022 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:20.645026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:20.645029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:20.645066 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:20.646851 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.647415 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.647419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.647421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.647542 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.647895 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.647900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.647902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.648139 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:20.648244 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.648968 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.649086 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 03:10:20.649799 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 22644
I20260929 03:10:20.649953 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:20.650223 21177 external_mini_cluster.cc:1428] Setting key 81287d793cc4bfc450fea67947593c43
I20260929 03:10:20.650353 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.635913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:41111 (local address 127.20.174.67:46253)
0929 03:10:20.636389 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.636450 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.637333 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.638193 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.638200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.638668 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:20.639155 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.639168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.640121 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.640124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.641051 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.641062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.641811 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.641816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.641996 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.642712 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:20.642740 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:20.644814 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:20.646995 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.647001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.647018 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.647294 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.647623 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.647628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.647630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.647792 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.648247 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:20.648253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:20.648569 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.648809 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.649229 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":139,"threads_started":1}
I20260929 03:10:20.652164 22775 heartbeater.cc:347] Connected to a master server at 127.20.174.126:41111
I20260929 03:10:20.652328 22775 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:20.652918 22775 heartbeater.cc:511] Master 127.20.174.126:41111 requested a full tablet report, sending...
I20260929 03:10:20.654019 22292 ts_manager.cc:194] Registered new tserver with Master: 21f5c489f32346668e7cb987eeecd5de (127.20.174.67:36881)
I20260929 03:10:20.654764 22292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:46253
I20260929 03:10:20.661896 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.654502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:58419 (local address 127.20.174.126:41111)
0929 03:10:20.654646 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.654651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.654884 (+ 233us) server_negotiation.cc:407] Connection header received
0929 03:10:20.655036 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.655039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.655082 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.655175 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:20.655679 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.656277 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.657189 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.657427 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.657543 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.658811 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:20.658815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:20.658817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:20.658855 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:20.660188 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.660645 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.660649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.660651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.660736 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.661047 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.661050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.661052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.661230 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:20.661344 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.661620 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.661745 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:10:20.661909 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.654403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:41111 (local address 127.20.174.67:58419)
0929 03:10:20.654790 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.654802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.654927 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.655191 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.655193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.655453 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:20.655556 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.655563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.656412 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.656415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.657068 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.657078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.657546 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.657550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.657640 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.658186 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:20.658197 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:20.658690 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:20.660316 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.660320 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.660323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.660533 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.660841 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:20.660845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:20.660851 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:20.660942 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:20.661338 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:20.661341 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:20.661443 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.661554 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.661715 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":91,"threads_started":1}
I20260929 03:10:20.664959 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:10:20 dist-test-slave-k1dg krb5kdc[22247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651420, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:10:20 dist-test-slave-k1dg krb5kdc[22247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651420, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:20.692760 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.683745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.20.174.126:41111)
0929 03:10:20.683888 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.683892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.683904 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:10:20.683944 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.683946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.683984 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.684097 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:20.684876 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.685410 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.686468 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.686712 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.686826 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.689524 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:20.689527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:20.689529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:20.689557 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:20.690967 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.691454 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.691458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.691460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.691544 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:20.691850 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:20.691853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:20.691855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:20.692046 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:20.692184 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.692463 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.692593 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:10:20.696029 22292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:20.699167 22292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:20.715535 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.708760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:34615 (local address 127.0.0.1:51996)
0929 03:10:20.709577 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.709590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.709670 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.710990 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.710997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.711023 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:20.711122 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.711128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.712575 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.712579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.713955 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.713969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.715064 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.715070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.715202 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.715225 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.715287 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.715357 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":62,"thread_start_us":119,"threads_started":1}
I20260929 03:10:20.715600 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.710172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.20.174.66:34615)
0929 03:10:20.710552 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.710564 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.710585 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:10:20.710654 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.710660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.710831 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.711000 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:20.711257 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.712450 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.714121 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.714941 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.715064 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.715248 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.715326 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.715432 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":123,"threads_started":1}
I20260929 03:10:20.715727 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.709782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.20.174.65:40431)
0929 03:10:20.710229 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.710234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.710252 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:10:20.710309 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.710341 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.710621 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.711058 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:20.711227 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.712707 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.713799 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.714668 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.714781 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.714939 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.715447 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.715559 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":99,"threads_started":1}
I20260929 03:10:20.716496 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.708579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:36881 (local address 127.0.0.1:57782)
0929 03:10:20.709079 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.709140 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.709298 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.710677 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.710684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.710710 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:20.710890 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.710905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.712217 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.712225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.714000 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.714009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.715087 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.715090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.716200 (+ 1110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.716225 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.716291 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.716361 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":1071,"thread_start_us":74,"threads_started":1}
I20260929 03:10:20.716758 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.709294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:40431 (local address 127.0.0.1:56988)
0929 03:10:20.709913 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.709926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.710014 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.710770 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.710773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.710791 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:20.710920 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.710928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.712828 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.712831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.713661 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.713670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.714817 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.714822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.714947 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.715010 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.715132 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.715282 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":134,"threads_started":1}
I20260929 03:10:20.717133 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.709377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.20.174.67:36881)
0929 03:10:20.710052 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.710066 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.710093 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:10:20.710179 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.710184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.710473 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.710692 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:20.711099 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.712066 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.714137 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.714957 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.715226 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.715658 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.716913 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.717023 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":105,"threads_started":1}
I20260929 03:10:20.719635 22569 tablet_service.cc:1467] Processing CreateTablet for tablet 12abdd5cb94c4d34a25d3ac165e9f69b (DEFAULT_TABLE table=test-table [id=63d960f9f3ff40fb877fb63a735b4258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:20.720028 22709 tablet_service.cc:1467] Processing CreateTablet for tablet 12abdd5cb94c4d34a25d3ac165e9f69b (DEFAULT_TABLE table=test-table [id=63d960f9f3ff40fb877fb63a735b4258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:20.721148 22709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12abdd5cb94c4d34a25d3ac165e9f69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:20.720974 22429 tablet_service.cc:1467] Processing CreateTablet for tablet 12abdd5cb94c4d34a25d3ac165e9f69b (DEFAULT_TABLE table=test-table [id=63d960f9f3ff40fb877fb63a735b4258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:20.721926 22429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12abdd5cb94c4d34a25d3ac165e9f69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:20.722105 22569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12abdd5cb94c4d34a25d3ac165e9f69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:20.728592 22798 tablet_bootstrap.cc:492] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74: Bootstrap starting.
I20260929 03:10:20.730798 22799 tablet_bootstrap.cc:492] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de: Bootstrap starting.
I20260929 03:10:20.731362 22798 tablet_bootstrap.cc:654] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:20.731580 22800 tablet_bootstrap.cc:492] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04: Bootstrap starting.
I20260929 03:10:20.732580 22798 log.cc:824] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:20.733384 22799 tablet_bootstrap.cc:654] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:20.734004 22800 tablet_bootstrap.cc:654] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:20.734392 22799 log.cc:824] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:20.734790 22800 log.cc:824] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:20.735069 22798 tablet_bootstrap.cc:492] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74: No bootstrap required, opened a new log
I20260929 03:10:20.735339 22798 ts_tablet_manager.cc:1397] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 03:10:20.736661 22799 tablet_bootstrap.cc:492] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de: No bootstrap required, opened a new log
I20260929 03:10:20.737015 22800 tablet_bootstrap.cc:492] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04: No bootstrap required, opened a new log
I20260929 03:10:20.737093 22799 ts_tablet_manager.cc:1397] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:10:20.737226 22800 ts_tablet_manager.cc:1397] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:10:20.740576 22800 raft_consensus.cc:348] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } }
I20260929 03:10:20.740846 22800 raft_consensus.cc:374] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:20.740655 22798 raft_consensus.cc:348] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } }
I20260929 03:10:20.740936 22800 raft_consensus.cc:729] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39897749def34c9f97fbcdf8570d9c04, State: Initialized, Role: FOLLOWER
I20260929 03:10:20.740935 22798 raft_consensus.cc:374] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:20.740994 22798 raft_consensus.cc:729] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2815b56ae26f48cb91a29df1f0794f74, State: Initialized, Role: FOLLOWER
I20260929 03:10:20.741442 22800 consensus_queue.cc:260] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } }
I20260929 03:10:20.741365 22799 raft_consensus.cc:348] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } }
I20260929 03:10:20.741526 22798 consensus_queue.cc:260] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } }
I20260929 03:10:20.741617 22799 raft_consensus.cc:374] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:20.741680 22799 raft_consensus.cc:729] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f5c489f32346668e7cb987eeecd5de, State: Initialized, Role: FOLLOWER
I20260929 03:10:20.742192 22799 consensus_queue.cc:260] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } }
I20260929 03:10:20.742273 22495 heartbeater.cc:503] Master 127.20.174.126:41111 was elected leader, sending a full tablet report...
I20260929 03:10:20.742614 22800 ts_tablet_manager.cc:1428] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260929 03:10:20.742692 22798 ts_tablet_manager.cc:1428] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 03:10:20.743041 22635 heartbeater.cc:503] Master 127.20.174.126:41111 was elected leader, sending a full tablet report...
I20260929 03:10:20.743326 22799 ts_tablet_manager.cc:1428] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:10:20.744221 22775 heartbeater.cc:503] Master 127.20.174.126:41111 was elected leader, sending a full tablet report...
I20260929 03:10:20.798147 22804 raft_consensus.cc:482] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:20.798460 22804 raft_consensus.cc:504] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } }
I20260929 03:10:20.799717 22804 leader_election.cc:288] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21f5c489f32346668e7cb987eeecd5de (127.20.174.67:36881), 2815b56ae26f48cb91a29df1f0794f74 (127.20.174.65:40431)
I20260929 03:10:20.805156 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.800074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:40431 (local address 127.20.174.66:37869)
0929 03:10:20.800240 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.800269 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.800387 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.800846 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.800848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.800862 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:20.800960 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.800964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.802144 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.802148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.803836 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.803848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.804548 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.804552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.804772 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.804796 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.804889 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.804998 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":7}
I20260929 03:10:20.805141 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.800204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:37869 (local address 127.20.174.65:40431)
0929 03:10:20.800351 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.800356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.800371 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:20.800534 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.800539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.800591 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.800694 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:20.801118 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.801949 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.803970 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.804408 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.804554 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.804611 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.804910 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.804997 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:10:20.805141 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.800365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:36935 (local address 127.20.174.67:36881)
0929 03:10:20.800499 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.800504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.800664 (+ 160us) server_negotiation.cc:407] Connection header received
0929 03:10:20.800890 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.800893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.800940 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.801001 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:20.801285 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.802040 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.803919 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.804392 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.804546 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.804611 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.804906 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.805003 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:10:20.805173 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.800074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:36881 (local address 127.20.174.66:36935)
0929 03:10:20.800560 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.800572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.800698 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.801048 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.801052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.801071 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:20.801177 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.801182 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.802155 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.802158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.803650 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.803663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.804542 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.804546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.804729 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.804784 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.804888 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.805002 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":259,"thread_start_us":139,"threads_started":1}
I20260929 03:10:20.806155 22449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12abdd5cb94c4d34a25d3ac165e9f69b" candidate_uuid: "39897749def34c9f97fbcdf8570d9c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2815b56ae26f48cb91a29df1f0794f74" is_pre_election: true
I20260929 03:10:20.806157 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12abdd5cb94c4d34a25d3ac165e9f69b" candidate_uuid: "39897749def34c9f97fbcdf8570d9c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f5c489f32346668e7cb987eeecd5de" is_pre_election: true
I20260929 03:10:20.806529 22449 raft_consensus.cc:2454] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39897749def34c9f97fbcdf8570d9c04 in term 0.
I20260929 03:10:20.806528 22729 raft_consensus.cc:2454] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39897749def34c9f97fbcdf8570d9c04 in term 0.
I20260929 03:10:20.807052 22522 leader_election.cc:302] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21f5c489f32346668e7cb987eeecd5de, 2815b56ae26f48cb91a29df1f0794f74, 39897749def34c9f97fbcdf8570d9c04; no voters:
I20260929 03:10:20.807341 22804 raft_consensus.cc:2781] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:20.807448 22804 raft_consensus.cc:482] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:20.807511 22804 raft_consensus.cc:3037] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:20.808614 22804 raft_consensus.cc:504] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } }
I20260929 03:10:20.808997 22804 leader_election.cc:288] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [CANDIDATE]: Term 1 election: Requested vote from peers 21f5c489f32346668e7cb987eeecd5de (127.20.174.67:36881), 2815b56ae26f48cb91a29df1f0794f74 (127.20.174.65:40431)
I20260929 03:10:20.809388 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12abdd5cb94c4d34a25d3ac165e9f69b" candidate_uuid: "39897749def34c9f97fbcdf8570d9c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f5c489f32346668e7cb987eeecd5de"
I20260929 03:10:20.809489 22449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12abdd5cb94c4d34a25d3ac165e9f69b" candidate_uuid: "39897749def34c9f97fbcdf8570d9c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2815b56ae26f48cb91a29df1f0794f74"
I20260929 03:10:20.809538 22729 raft_consensus.cc:3037] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:20.809671 22449 raft_consensus.cc:3037] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:20.810986 22449 raft_consensus.cc:2454] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39897749def34c9f97fbcdf8570d9c04 in term 1.
I20260929 03:10:20.811004 22729 raft_consensus.cc:2454] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39897749def34c9f97fbcdf8570d9c04 in term 1.
I20260929 03:10:20.811335 22522 leader_election.cc:302] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21f5c489f32346668e7cb987eeecd5de, 2815b56ae26f48cb91a29df1f0794f74, 39897749def34c9f97fbcdf8570d9c04; no voters:
I20260929 03:10:20.811578 22804 raft_consensus.cc:2781] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:20.811757 22804 raft_consensus.cc:686] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 1 LEADER]: Becoming Leader. State: Replica: 39897749def34c9f97fbcdf8570d9c04, State: Running, Role: LEADER
I20260929 03:10:20.812144 22804 consensus_queue.cc:237] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } }
I20260929 03:10:20.815630 22292 catalog_manager.cc:5495] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39897749def34c9f97fbcdf8570d9c04 (127.20.174.66). New cstate: current_term: 1 leader_uuid: "39897749def34c9f97fbcdf8570d9c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } health_report { overall_health: UNKNOWN } } }
I20260929 03:10:20.845762 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.841203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.20.174.66:34615)
0929 03:10:20.841345 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.841349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.841364 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:20.841571 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.841574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.841658 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.841758 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:20.842077 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.842656 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.843705 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.844027 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.844127 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.844700 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:20.845227 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:20.845334 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.845563 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.845633 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:10:20.853379 22449 raft_consensus.cc:1261] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74 [term 1 FOLLOWER]: Refusing update from remote peer 39897749def34c9f97fbcdf8570d9c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:20.853401 22729 raft_consensus.cc:1261] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de [term 1 FOLLOWER]: Refusing update from remote peer 39897749def34c9f97fbcdf8570d9c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:20.854246 22804 consensus_queue.cc:1035] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:20.854521 22808 consensus_queue.cc:1035] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:20.864095 22813 mvcc.cc:205] Tried to move back new op lower bound from 7334508219806031872 to 7334508219649314816. Current Snapshot: MvccSnapshot[applied={T|T < 7334508219806031872}]
I20260929 03:10:20.879182 22290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:20.879391 22290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:20.882277 22290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91: Sending DeleteTablet for 3 replicas of tablet 12abdd5cb94c4d34a25d3ac165e9f69b
I20260929 03:10:20.883328 22709 tablet_service.cc:1514] Processing DeleteTablet for tablet 12abdd5cb94c4d34a25d3ac165e9f69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:20 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:57782
I20260929 03:10:20.883600 22429 tablet_service.cc:1514] Processing DeleteTablet for tablet 12abdd5cb94c4d34a25d3ac165e9f69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:20 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:56988
I20260929 03:10:20.884080 22822 tablet_replica.cc:330] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74: stopping tablet replica
I20260929 03:10:20.884377 22569 tablet_service.cc:1514] Processing DeleteTablet for tablet 12abdd5cb94c4d34a25d3ac165e9f69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:20 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:51996
I20260929 03:10:20.884502 22822 raft_consensus.cc:2229] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:20.884845 22822 raft_consensus.cc:2258] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:20.886132 22290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:10:20.886771 22822 ts_tablet_manager.cc:1905] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:10:20.886893 22290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:20.889917 22822 ts_tablet_manager.cc:1918] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:20.890120 22822 log.cc:1196] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/wal/wals/12abdd5cb94c4d34a25d3ac165e9f69b
I20260929 03:10:20.890576 22822 ts_tablet_manager.cc:1939] T 12abdd5cb94c4d34a25d3ac165e9f69b P 2815b56ae26f48cb91a29df1f0794f74: Deleting consensus metadata
I20260929 03:10:20.892076 22279 catalog_manager.cc:4841] TS 2815b56ae26f48cb91a29df1f0794f74 (127.20.174.65:40431): tablet 12abdd5cb94c4d34a25d3ac165e9f69b (table test-table [id=63d960f9f3ff40fb877fb63a735b4258]) successfully deleted
I20260929 03:10:20.892778 22429 tablet_service.cc:1467] Processing CreateTablet for tablet a21250a9725248afb6407abbd28eabd3 (DEFAULT_TABLE table=test-table [id=7dbecd58e6cb4c8287c3f0336e517207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:20.892817 22709 tablet_service.cc:1467] Processing CreateTablet for tablet a21250a9725248afb6407abbd28eabd3 (DEFAULT_TABLE table=test-table [id=7dbecd58e6cb4c8287c3f0336e517207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:20.893050 22709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21250a9725248afb6407abbd28eabd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:20.893049 22429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21250a9725248afb6407abbd28eabd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:20.894811 22823 tablet_replica.cc:330] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04: stopping tablet replica
I20260929 03:10:20.894884 22569 tablet_service.cc:1467] Processing CreateTablet for tablet a21250a9725248afb6407abbd28eabd3 (DEFAULT_TABLE table=test-table [id=7dbecd58e6cb4c8287c3f0336e517207]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:20.895157 22569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21250a9725248afb6407abbd28eabd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:20.895344 22821 tablet_replica.cc:330] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de: stopping tablet replica
I20260929 03:10:20.895617 22821 raft_consensus.cc:2229] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:20.895994 22821 raft_consensus.cc:2258] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:20.896353 22798 tablet_bootstrap.cc:492] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74: Bootstrap starting.
I20260929 03:10:20.897387 22798 tablet_bootstrap.cc:654] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:20.897680 22800 tablet_bootstrap.cc:492] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04: Bootstrap starting.
I20260929 03:10:20.897969 22821 ts_tablet_manager.cc:1905] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:20.899317 22799 tablet_bootstrap.cc:492] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de: Bootstrap starting.
I20260929 03:10:20.899742 22800 tablet_bootstrap.cc:654] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:20.900302 22799 tablet_bootstrap.cc:654] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:20.895166 22823 raft_consensus.cc:2229] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:20.901127 22821 ts_tablet_manager.cc:1918] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:20.901288 22821 log.cc:1196] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/12abdd5cb94c4d34a25d3ac165e9f69b
I20260929 03:10:20.901460 22823 raft_consensus.cc:2258] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:20.901662 22821 ts_tablet_manager.cc:1939] T 12abdd5cb94c4d34a25d3ac165e9f69b P 21f5c489f32346668e7cb987eeecd5de: Deleting consensus metadata
I20260929 03:10:20.902318 22800 tablet_bootstrap.cc:492] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04: No bootstrap required, opened a new log
I20260929 03:10:20.902405 22800 ts_tablet_manager.cc:1397] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 03:10:20.902480 22799 tablet_bootstrap.cc:492] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de: No bootstrap required, opened a new log
I20260929 03:10:20.902561 22799 ts_tablet_manager.cc:1397] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:20.902894 22800 raft_consensus.cc:348] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:20.903013 22800 raft_consensus.cc:374] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:20.903053 22800 raft_consensus.cc:729] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39897749def34c9f97fbcdf8570d9c04, State: Initialized, Role: FOLLOWER
I20260929 03:10:20.903021 22799 raft_consensus.cc:348] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:20.903149 22799 raft_consensus.cc:374] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:20.903184 22799 raft_consensus.cc:729] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f5c489f32346668e7cb987eeecd5de, State: Initialized, Role: FOLLOWER
I20260929 03:10:20.903203 22823 ts_tablet_manager.cc:1905] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:20.903190 22800 consensus_queue.cc:260] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:20.903326 22799 consensus_queue.cc:260] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:20.903509 22800 ts_tablet_manager.cc:1428] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:20.903595 22799 ts_tablet_manager.cc:1428] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:20.903947 22278 catalog_manager.cc:4841] TS 21f5c489f32346668e7cb987eeecd5de (127.20.174.67:36881): tablet 12abdd5cb94c4d34a25d3ac165e9f69b (table test-table [id=63d960f9f3ff40fb877fb63a735b4258]) successfully deleted
I20260929 03:10:20.906497 22798 tablet_bootstrap.cc:492] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74: No bootstrap required, opened a new log
I20260929 03:10:20.906613 22798 ts_tablet_manager.cc:1397] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260929 03:10:20.906978 22823 ts_tablet_manager.cc:1918] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:20.907086 22823 log.cc:1196] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/12abdd5cb94c4d34a25d3ac165e9f69b
I20260929 03:10:20.907155 22798 raft_consensus.cc:348] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:20.907275 22798 raft_consensus.cc:374] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:20.907310 22798 raft_consensus.cc:729] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2815b56ae26f48cb91a29df1f0794f74, State: Initialized, Role: FOLLOWER
I20260929 03:10:20.907423 22823 ts_tablet_manager.cc:1939] T 12abdd5cb94c4d34a25d3ac165e9f69b P 39897749def34c9f97fbcdf8570d9c04: Deleting consensus metadata
I20260929 03:10:20.908384 22798 consensus_queue.cc:260] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:20.908967 22798 ts_tablet_manager.cc:1428] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:10:20.908952 22276 catalog_manager.cc:4841] TS 39897749def34c9f97fbcdf8570d9c04 (127.20.174.66:34615): tablet 12abdd5cb94c4d34a25d3ac165e9f69b (table test-table [id=63d960f9f3ff40fb877fb63a735b4258]) successfully deleted
W20260929 03:10:20.914477 22636 tablet.cc:2369] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:10:20.931442 22496 tablet.cc:2369] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:10:20.957106 22806 raft_consensus.cc:482] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:20.957280 22806 raft_consensus.cc:504] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:20.958684 22806 leader_election.cc:288] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2815b56ae26f48cb91a29df1f0794f74 (127.20.174.65:40431), 39897749def34c9f97fbcdf8570d9c04 (127.20.174.66:34615)
I20260929 03:10:20.963670 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.959264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:41917 (local address 127.20.174.66:34615)
0929 03:10:20.959408 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.959413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.959609 (+ 196us) server_negotiation.cc:407] Connection header received
0929 03:10:20.959811 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.959814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.959910 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.959997 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:20.960262 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.960744 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.962261 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.962971 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.963105 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.963185 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.963449 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.963538 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:10:20.963670 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.958934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:34615 (local address 127.20.174.67:41917)
0929 03:10:20.959518 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.959531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.959701 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.960018 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.960021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.960036 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:20.960151 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.960156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.960872 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.960878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.962100 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.962113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.963133 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.963137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.963240 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.963326 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.963410 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.963488 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":121,"spinlock_wait_cycles":27776,"thread_start_us":74,"threads_started":1}
I20260929 03:10:20.963975 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.959169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:37553 (local address 127.20.174.65:40431)
0929 03:10:20.959304 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:20.959309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:20.959324 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:20.959488 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:20.959491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:20.959533 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:20.959624 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:20.959978 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.960777 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.962403 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.963103 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.963211 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.963261 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:20.963732 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 03:10:20.963821 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:10:20.964419 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:20.958894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:40431 (local address 127.20.174.67:37553)
0929 03:10:20.959208 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:20.959223 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:20.959319 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:20.959651 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:20.959653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:20.959699 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:20.959828 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.959836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.960917 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.960921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.962247 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:20.962259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:20.963260 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:20.963264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:20.963451 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:20.963483 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:10:20.963557 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:10:20.963629 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":66,"spinlock_wait_cycles":4096}
I20260929 03:10:20.964704 22589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21250a9725248afb6407abbd28eabd3" candidate_uuid: "21f5c489f32346668e7cb987eeecd5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39897749def34c9f97fbcdf8570d9c04" is_pre_election: true
I20260929 03:10:20.964879 22449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21250a9725248afb6407abbd28eabd3" candidate_uuid: "21f5c489f32346668e7cb987eeecd5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2815b56ae26f48cb91a29df1f0794f74" is_pre_election: true
I20260929 03:10:20.965008 22589 raft_consensus.cc:2454] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f5c489f32346668e7cb987eeecd5de in term 0.
I20260929 03:10:20.965032 22449 raft_consensus.cc:2454] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f5c489f32346668e7cb987eeecd5de in term 0.
I20260929 03:10:20.965458 22663 leader_election.cc:302] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f5c489f32346668e7cb987eeecd5de, 2815b56ae26f48cb91a29df1f0794f74; no voters:
I20260929 03:10:20.965837 22806 raft_consensus.cc:2781] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:20.965920 22806 raft_consensus.cc:482] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:20.965973 22806 raft_consensus.cc:3037] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:20.967079 22806 raft_consensus.cc:504] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:20.967517 22806 leader_election.cc:288] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [CANDIDATE]: Term 1 election: Requested vote from peers 2815b56ae26f48cb91a29df1f0794f74 (127.20.174.65:40431), 39897749def34c9f97fbcdf8570d9c04 (127.20.174.66:34615)
I20260929 03:10:20.967941 22449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21250a9725248afb6407abbd28eabd3" candidate_uuid: "21f5c489f32346668e7cb987eeecd5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2815b56ae26f48cb91a29df1f0794f74"
I20260929 03:10:20.967948 22589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21250a9725248afb6407abbd28eabd3" candidate_uuid: "21f5c489f32346668e7cb987eeecd5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39897749def34c9f97fbcdf8570d9c04"
I20260929 03:10:20.968084 22589 raft_consensus.cc:3037] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:20.968091 22449 raft_consensus.cc:3037] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:20.969049 22449 raft_consensus.cc:2454] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f5c489f32346668e7cb987eeecd5de in term 1.
I20260929 03:10:20.969272 22589 raft_consensus.cc:2454] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f5c489f32346668e7cb987eeecd5de in term 1.
I20260929 03:10:20.969372 22663 leader_election.cc:302] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f5c489f32346668e7cb987eeecd5de, 2815b56ae26f48cb91a29df1f0794f74; no voters:
I20260929 03:10:20.969554 22806 raft_consensus.cc:2781] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:20.969796 22806 raft_consensus.cc:686] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 1 LEADER]: Becoming Leader. State: Replica: 21f5c489f32346668e7cb987eeecd5de, State: Running, Role: LEADER
I20260929 03:10:20.970057 22806 consensus_queue.cc:237] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:20.972558 22290 catalog_manager.cc:5495] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f5c489f32346668e7cb987eeecd5de (127.20.174.67). New cstate: current_term: 1 leader_uuid: "21f5c489f32346668e7cb987eeecd5de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } health_report { overall_health: HEALTHY } } }
Sep 29 03:10:21 dist-test-slave-k1dg krb5kdc[22247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651419, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:21.023702 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.012633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:41111 (local address 127.0.0.1:41086)
0929 03:10:21.013115 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:21.013130 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:21.013271 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:21.013607 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:21.013610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:21.014050 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:21.014177 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.014183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.015313 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.015316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.015934 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.015944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.016611 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.016614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.016681 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.017458 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:21.017502 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:21.019226 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:21.021717 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:21.021731 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:21.021737 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:21.022122 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:21.022498 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:21.022501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:21.022503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:21.022599 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:21.023096 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:21.023103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:21.023230 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 03:10:21.023339 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:10:21.023557 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":130,"threads_started":1}
I20260929 03:10:21.023838 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.012715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.20.174.126:41111)
0929 03:10:21.012868 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:21.012874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:21.013255 (+ 381us) server_negotiation.cc:407] Connection header received
0929 03:10:21.013383 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:21.013387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:21.013446 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:21.013547 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:21.014348 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.015161 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.016095 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.016463 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.016612 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.019421 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:21.019425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:21.019428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:21.019459 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:21.021564 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:21.022216 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:21.022220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:21.022222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:21.022338 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:21.022703 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:21.022706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:21.022708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:21.022968 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:21.023095 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:21.023442 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 03:10:21.023644 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":52}
I20260929 03:10:21.029572 22290 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:41086:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:10:21.030553 22290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:21.036365 22569 tablet_service.cc:1467] Processing CreateTablet for tablet 3b157170219242579ee351bd342f9973 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4b4ff33e40349f8b7a615d8ffb752b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:10:21.036293 22429 tablet_service.cc:1467] Processing CreateTablet for tablet 3b157170219242579ee351bd342f9973 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4b4ff33e40349f8b7a615d8ffb752b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:10:21.036451 22709 tablet_service.cc:1467] Processing CreateTablet for tablet 3b157170219242579ee351bd342f9973 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4b4ff33e40349f8b7a615d8ffb752b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:10:21.036748 22569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b157170219242579ee351bd342f9973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:21.036751 22429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b157170219242579ee351bd342f9973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:21.036836 22709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b157170219242579ee351bd342f9973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:21.040059 22798 tablet_bootstrap.cc:492] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: Bootstrap starting.
I20260929 03:10:21.040059 22800 tablet_bootstrap.cc:492] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04: Bootstrap starting.
I20260929 03:10:21.040084 22799 tablet_bootstrap.cc:492] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de: Bootstrap starting.
I20260929 03:10:21.041131 22799 tablet_bootstrap.cc:654] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:21.041224 22798 tablet_bootstrap.cc:654] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:21.041653 22800 tablet_bootstrap.cc:654] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:21.043190 22799 tablet_bootstrap.cc:492] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de: No bootstrap required, opened a new log
I20260929 03:10:21.043208 22800 tablet_bootstrap.cc:492] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04: No bootstrap required, opened a new log
I20260929 03:10:21.043285 22799 ts_tablet_manager.cc:1397] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:21.043300 22800 ts_tablet_manager.cc:1397] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:21.043850 22800 raft_consensus.cc:348] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:21.043951 22798 tablet_bootstrap.cc:492] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: No bootstrap required, opened a new log
I20260929 03:10:21.043977 22800 raft_consensus.cc:374] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:21.044010 22800 raft_consensus.cc:729] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39897749def34c9f97fbcdf8570d9c04, State: Initialized, Role: FOLLOWER
I20260929 03:10:21.044056 22798 ts_tablet_manager.cc:1397] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:10:21.044031 22799 raft_consensus.cc:348] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:21.044169 22799 raft_consensus.cc:374] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:21.044212 22799 raft_consensus.cc:729] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f5c489f32346668e7cb987eeecd5de, State: Initialized, Role: FOLLOWER
I20260929 03:10:21.044193 22800 consensus_queue.cc:260] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:21.044359 22799 consensus_queue.cc:260] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:21.044433 22816 tablet_replica.cc:439] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } }
I20260929 03:10:21.044579 22816 tablet_replica.cc:442] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:21.044535 22798 raft_consensus.cc:348] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:21.044833 22798 raft_consensus.cc:374] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:21.044875 22798 raft_consensus.cc:729] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2815b56ae26f48cb91a29df1f0794f74, State: Initialized, Role: FOLLOWER
I20260929 03:10:21.044981 22806 tablet_replica.cc:439] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } }
I20260929 03:10:21.045015 22798 consensus_queue.cc:260] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:21.045104 22806 tablet_replica.cc:442] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:21.045325 22798 ts_tablet_manager.cc:1428] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:21.045567 22805 tablet_replica.cc:439] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } }
I20260929 03:10:21.045698 22805 tablet_replica.cc:442] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:21.046795 22800 ts_tablet_manager.cc:1428] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:21.044625 22799 ts_tablet_manager.cc:1428] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:21.064054 22805 raft_consensus.cc:482] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:21.064248 22805 raft_consensus.cc:504] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:21.065634 22805 leader_election.cc:288] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39897749def34c9f97fbcdf8570d9c04 (127.20.174.66:34615), 21f5c489f32346668e7cb987eeecd5de (127.20.174.67:36881)
I20260929 03:10:21.069943 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.066197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:48667 (local address 127.20.174.67:36881)
0929 03:10:21.066349 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:21.066353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:21.066507 (+ 154us) server_negotiation.cc:407] Connection header received
0929 03:10:21.066677 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:21.066685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:21.066724 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:21.066814 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:21.067178 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.067652 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.068894 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.069287 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.069428 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.069490 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:21.069737 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 03:10:21.069805 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:10:21.069960 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.065861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:34615 (local address 127.20.174.65:49047)
0929 03:10:21.066350 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:21.066362 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:21.066528 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:21.066817 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:21.066820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:21.066835 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:21.067005 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.067012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.067783 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.067788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.068711 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.068720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.069436 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.069440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.069579 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.069600 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:10:21.069661 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:10:21.069764 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":56,"thread_start_us":160,"threads_started":1}
I20260929 03:10:21.069941 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.065967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:49047 (local address 127.20.174.66:34615)
0929 03:10:21.066127 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:21.066133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:21.066450 (+ 317us) server_negotiation.cc:407] Connection header received
0929 03:10:21.066668 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:21.066671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:21.066719 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:21.066778 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:21.067118 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.067644 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.068854 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.069283 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.069426 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.069487 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:21.069706 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 03:10:21.069808 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:10:21.069971 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.065859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:36881 (local address 127.20.174.65:48667)
0929 03:10:21.066381 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:21.066396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:21.066549 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:21.066854 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:21.066857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:21.066881 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:21.067039 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.067045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.067789 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.067795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.068724 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.068734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.069427 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.069431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.069588 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.069608 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:10:21.069704 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 03:10:21.069808 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":88,"spinlock_wait_cycles":3200,"thread_start_us":171,"threads_started":1}
I20260929 03:10:21.070518 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b157170219242579ee351bd342f9973" candidate_uuid: "2815b56ae26f48cb91a29df1f0794f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f5c489f32346668e7cb987eeecd5de" is_pre_election: true
I20260929 03:10:21.070529 22589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b157170219242579ee351bd342f9973" candidate_uuid: "2815b56ae26f48cb91a29df1f0794f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39897749def34c9f97fbcdf8570d9c04" is_pre_election: true
I20260929 03:10:21.070667 22729 raft_consensus.cc:2454] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2815b56ae26f48cb91a29df1f0794f74 in term 0.
I20260929 03:10:21.070698 22589 raft_consensus.cc:2454] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2815b56ae26f48cb91a29df1f0794f74 in term 0.
I20260929 03:10:21.071043 22382 leader_election.cc:302] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f5c489f32346668e7cb987eeecd5de, 2815b56ae26f48cb91a29df1f0794f74; no voters:
I20260929 03:10:21.071388 22805 raft_consensus.cc:2781] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:21.071460 22805 raft_consensus.cc:482] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:21.071501 22805 raft_consensus.cc:3037] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:21.072525 22805 raft_consensus.cc:504] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:21.072837 22805 leader_election.cc:288] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [CANDIDATE]: Term 1 election: Requested vote from peers 39897749def34c9f97fbcdf8570d9c04 (127.20.174.66:34615), 21f5c489f32346668e7cb987eeecd5de (127.20.174.67:36881)
I20260929 03:10:21.073233 22589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b157170219242579ee351bd342f9973" candidate_uuid: "2815b56ae26f48cb91a29df1f0794f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39897749def34c9f97fbcdf8570d9c04"
I20260929 03:10:21.073371 22589 raft_consensus.cc:3037] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:21.074324 22589 raft_consensus.cc:2454] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2815b56ae26f48cb91a29df1f0794f74 in term 1.
I20260929 03:10:21.074630 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b157170219242579ee351bd342f9973" candidate_uuid: "2815b56ae26f48cb91a29df1f0794f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f5c489f32346668e7cb987eeecd5de"
I20260929 03:10:21.074767 22380 leader_election.cc:302] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2815b56ae26f48cb91a29df1f0794f74, 39897749def34c9f97fbcdf8570d9c04; no voters:
I20260929 03:10:21.074811 22729 raft_consensus.cc:3037] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:21.075800 22729 raft_consensus.cc:2454] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2815b56ae26f48cb91a29df1f0794f74 in term 1.
I20260929 03:10:21.076102 22805 raft_consensus.cc:2781] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:21.076354 22805 raft_consensus.cc:686] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [term 1 LEADER]: Becoming Leader. State: Replica: 2815b56ae26f48cb91a29df1f0794f74, State: Running, Role: LEADER
I20260929 03:10:21.076639 22805 consensus_queue.cc:237] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } }
I20260929 03:10:21.077943 22841 tablet_replica.cc:439] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: TxnStatusTablet state changed. Reason: New leader 2815b56ae26f48cb91a29df1f0794f74. Latest consensus state: current_term: 1 leader_uuid: "2815b56ae26f48cb91a29df1f0794f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } }
I20260929 03:10:21.078074 22841 tablet_replica.cc:442] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:10:21.078630 22843 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:10:21.078796 22843 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:10:21.079255 22291 catalog_manager.cc:5495] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2815b56ae26f48cb91a29df1f0794f74 (127.20.174.65). New cstate: current_term: 1 leader_uuid: "2815b56ae26f48cb91a29df1f0794f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } health_report { overall_health: UNKNOWN } } }
I20260929 03:10:21.099265 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.094092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:40431 (local address 127.0.0.1:57004)
0929 03:10:21.094306 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:21.094325 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:21.094497 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:21.095080 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:21.095083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:21.095091 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:10:21.095203 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.095208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.096304 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.096307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.097152 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.097163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.097885 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.097889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.097971 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.098029 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:10:21.098974 (+ 945us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:10:21.098991 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 03:10:21.099061 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:10:21.099142 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 03:10:21.099306 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.094216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.20.174.65:40431)
0929 03:10:21.094344 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:21.094348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:21.094371 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:10:21.094676 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:21.094680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:21.094751 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:21.094865 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:21.095341 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.096160 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.097306 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.097663 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.097770 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.098190 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:21.098810 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:21.098930 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:21.099101 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 03:10:21.099166 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:10:21.102852 22729 raft_consensus.cc:1261] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de [term 1 FOLLOWER]: Refusing update from remote peer 2815b56ae26f48cb91a29df1f0794f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:21.103094 22589 raft_consensus.cc:1261] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04 [term 1 FOLLOWER]: Refusing update from remote peer 2815b56ae26f48cb91a29df1f0794f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:21.103381 22841 consensus_queue.cc:1035] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:21.103737 22805 consensus_queue.cc:1035] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:21.105556 22816 tablet_replica.cc:439] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04: TxnStatusTablet state changed. Reason: New leader 2815b56ae26f48cb91a29df1f0794f74. Latest consensus state: current_term: 1 leader_uuid: "2815b56ae26f48cb91a29df1f0794f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } }
I20260929 03:10:21.105684 22816 tablet_replica.cc:442] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:21.105845 22806 tablet_replica.cc:439] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de: TxnStatusTablet state changed. Reason: New leader 2815b56ae26f48cb91a29df1f0794f74. Latest consensus state: current_term: 1 leader_uuid: "2815b56ae26f48cb91a29df1f0794f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } }
I20260929 03:10:21.105973 22806 tablet_replica.cc:442] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:21.107278 22806 tablet_replica.cc:439] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2815b56ae26f48cb91a29df1f0794f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } }
I20260929 03:10:21.107424 22806 tablet_replica.cc:442] T 3b157170219242579ee351bd342f9973 P 21f5c489f32346668e7cb987eeecd5de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:21.108006 22805 tablet_replica.cc:439] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2815b56ae26f48cb91a29df1f0794f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } }
I20260929 03:10:21.108119 22805 tablet_replica.cc:442] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:10:21.109123 22812 mvcc.cc:205] Tried to move back new op lower bound from 7334508220831186944 to 7334508220732383232. Current Snapshot: MvccSnapshot[applied={T|T < 7334508220831186944 or (T in {7334508220831186944})}]
I20260929 03:10:21.111289 22841 tablet_replica.cc:439] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2815b56ae26f48cb91a29df1f0794f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } }
I20260929 03:10:21.111409 22841 tablet_replica.cc:442] T 3b157170219242579ee351bd342f9973 P 2815b56ae26f48cb91a29df1f0794f74: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:10:21.112460 22816 tablet_replica.cc:439] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2815b56ae26f48cb91a29df1f0794f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 } } peers { permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 } } peers { permanent_uuid: "21f5c489f32346668e7cb987eeecd5de" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36881 } } }
I20260929 03:10:21.112589 22816 tablet_replica.cc:442] T 3b157170219242579ee351bd342f9973 P 39897749def34c9f97fbcdf8570d9c04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:21.113637 22811 mvcc.cc:205] Tried to move back new op lower bound from 7334508220831186944 to 7334508220732383232. Current Snapshot: MvccSnapshot[applied={T|T < 7334508220831186944 or (T in {7334508220831186944})}]
I20260929 03:10:21.119146 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.113531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.20.174.67:36881)
0929 03:10:21.115180 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:21.115185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:21.115199 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:10:21.115284 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:21.115288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:21.115356 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:21.115472 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:21.115770 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.116297 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.117506 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.117727 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.117810 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.118142 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:21.118699 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:21.118833 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:21.118900 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 03:10:21.118961 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:10:21.131762 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.127883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:51125 (local address 127.20.174.65:40431)
0929 03:10:21.128024 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:21.128028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:21.128044 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:10:21.128137 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:21.128140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:21.128194 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:21.128283 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:21.128586 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.129387 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.130354 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.130745 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.130855 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.131210 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:21.131304 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:21.131377 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:21.131560 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 03:10:21.131618 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:10:21.131820 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.127768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:40431 (local address 127.20.174.67:51125)
0929 03:10:21.127941 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:21.127964 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:21.128051 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:21.128328 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:21.128330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:21.128338 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:10:21.128462 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.128467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.129521 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.129524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.130222 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.130232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.130908 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.130912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.130997 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.131049 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:10:21.131433 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:10:21.131441 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:10:21.131523 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:10:21.131604 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 03:10:21.139789 22449 raft_consensus.cc:1261] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74 [term 1 FOLLOWER]: Refusing update from remote peer 21f5c489f32346668e7cb987eeecd5de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:21.139851 22589 raft_consensus.cc:1261] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04 [term 1 FOLLOWER]: Refusing update from remote peer 21f5c489f32346668e7cb987eeecd5de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:21.140355 22806 consensus_queue.cc:1035] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "2815b56ae26f48cb91a29df1f0794f74" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:21.140609 22828 consensus_queue.cc:1035] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "39897749def34c9f97fbcdf8570d9c04" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 34615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:21.167126 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.162941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:58857 (local address 127.20.174.67:36881)
0929 03:10:21.163070 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:21.163075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:21.163400 (+ 325us) server_negotiation.cc:407] Connection header received
0929 03:10:21.163514 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:21.163517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:21.163578 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:21.163656 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:21.163977 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.164776 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.166051 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.166276 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.166358 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.166723 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:21.166795 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:21.166852 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:21.166962 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 03:10:21.167011 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:10:21.167299 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.162817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:36881 (local address 127.20.174.65:58857)
0929 03:10:21.163288 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:21.163303 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:21.163422 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:21.163690 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:21.163693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:21.163701 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:10:21.163831 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.163837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.164925 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.164928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.165926 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:21.165939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.166395 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.166399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:21.166531 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.166594 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:10:21.166886 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:10:21.166893 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:10:21.167034 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 03:10:21.167127 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":141,"threads_started":1}
I20260929 03:10:21.199636 22291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:21.199821 22291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:21.202219 22291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89d9e5c474ec4b43bc37e3bf39d20d91: Sending DeleteTablet for 3 replicas of tablet a21250a9725248afb6407abbd28eabd3
I20260929 03:10:21.202939 22429 tablet_service.cc:1514] Processing DeleteTablet for tablet a21250a9725248afb6407abbd28eabd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:21 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:56988
I20260929 03:10:21.203166 22822 tablet_replica.cc:330] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74: stopping tablet replica
I20260929 03:10:21.203284 22822 raft_consensus.cc:2229] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:21.203291 22709 tablet_service.cc:1514] Processing DeleteTablet for tablet a21250a9725248afb6407abbd28eabd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:21 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:57782
I20260929 03:10:21.203423 22822 raft_consensus.cc:2258] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:21.203660 22569 tablet_service.cc:1514] Processing DeleteTablet for tablet a21250a9725248afb6407abbd28eabd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:21 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:51996
I20260929 03:10:21.203881 22823 tablet_replica.cc:330] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04: stopping tablet replica
I20260929 03:10:21.203903 22821 tablet_replica.cc:330] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de: stopping tablet replica
I20260929 03:10:21.204010 22823 raft_consensus.cc:2229] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:21.204023 22821 raft_consensus.cc:2229] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:21.204125 22823 raft_consensus.cc:2258] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:21.204339 22821 raft_consensus.cc:2258] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:21.205289 22822 ts_tablet_manager.cc:1905] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:21.205289 22823 ts_tablet_manager.cc:1905] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:21.207293 22821 ts_tablet_manager.cc:1905] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:21.208482 22822 ts_tablet_manager.cc:1918] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:21.208563 22822 log.cc:1196] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-0/wal/wals/a21250a9725248afb6407abbd28eabd3
I20260929 03:10:21.208819 22822 ts_tablet_manager.cc:1939] T a21250a9725248afb6407abbd28eabd3 P 2815b56ae26f48cb91a29df1f0794f74: Deleting consensus metadata
I20260929 03:10:21.210142 22821 ts_tablet_manager.cc:1918] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:21.210224 22821 log.cc:1196] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/a21250a9725248afb6407abbd28eabd3
I20260929 03:10:21.210274 22279 catalog_manager.cc:4841] TS 2815b56ae26f48cb91a29df1f0794f74 (127.20.174.65:40431): tablet a21250a9725248afb6407abbd28eabd3 (table test-table [id=7dbecd58e6cb4c8287c3f0336e517207]) successfully deleted
I20260929 03:10:21.210525 22821 ts_tablet_manager.cc:1939] T a21250a9725248afb6407abbd28eabd3 P 21f5c489f32346668e7cb987eeecd5de: Deleting consensus metadata
I20260929 03:10:21.212148 22278 catalog_manager.cc:4841] TS 21f5c489f32346668e7cb987eeecd5de (127.20.174.67:36881): tablet a21250a9725248afb6407abbd28eabd3 (table test-table [id=7dbecd58e6cb4c8287c3f0336e517207]) successfully deleted
I20260929 03:10:21.213845 22823 ts_tablet_manager.cc:1918] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:21.214017 22823 log.cc:1196] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/a21250a9725248afb6407abbd28eabd3
I20260929 03:10:21.214336 22823 ts_tablet_manager.cc:1939] T a21250a9725248afb6407abbd28eabd3 P 39897749def34c9f97fbcdf8570d9c04: Deleting consensus metadata
I20260929 03:10:21.215301 22276 catalog_manager.cc:4841] TS 39897749def34c9f97fbcdf8570d9c04 (127.20.174.66:34615): tablet a21250a9725248afb6407abbd28eabd3 (table test-table [id=7dbecd58e6cb4c8287c3f0336e517207]) successfully deleted
Sep 29 03:10:21 dist-test-slave-k1dg krb5kdc[22247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651420, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:10:21.238345 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.212473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.20.174.65:40431)
0929 03:10:21.212601 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:21.212605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:21.215376 (+ 2771us) server_negotiation.cc:407] Connection header received
0929 03:10:21.215659 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:21.215663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:21.215714 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:21.215768 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:21.216825 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.217648 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.219138 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.219522 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.227799 (+ 8277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.232670 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:21.232689 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:21.232713 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:21.232763 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:21.235478 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:21.236184 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:21.236194 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:21.236198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:21.236330 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:21.236708 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:21.236713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:21.236718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:21.237116 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:21.237358 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:21.237875 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 03:10:21.238053 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 03:10:21.239451 22489 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57008
I20260929 03:10:21.259958 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:21.247026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.20.174.126:41111)
0929 03:10:21.247209 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:21.247215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:21.247914 (+ 699us) server_negotiation.cc:407] Connection header received
0929 03:10:21.248169 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:21.248176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:21.248244 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:21.248311 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:21.249558 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.250665 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.252390 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:21.252874 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:21.252990 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:21.255358 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:21.255366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:21.255382 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:21.255425 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:21.257477 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:21.258129 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:21.258135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:21.258138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:21.258238 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:21.258678 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:21.258684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:21.258689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:21.258977 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:21.259100 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:21.259537 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 03:10:21.259704 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 03:10:21.261353 22291 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41088
I20260929 03:10:21.264004 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 22364
I20260929 03:10:21.275621 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 22504
I20260929 03:10:21.283455 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 22644
I20260929 03:10:21.289881 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 22260
2026-09-29T03:10:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2856 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:21 dist-test-slave-k1dg krb5kdc[22864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:21 dist-test-slave-k1dg krb5kdc[22864](info): set up 2 sockets
Sep 29 03:10:21 dist-test-slave-k1dg krb5kdc[22864](info): commencing operation
krb5kdc: starting...
W20260929 03:10:22.346774 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:22 dist-test-slave-k1dg krb5kdc[22864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:22.483705 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44707
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:44707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:22.599092 22877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:22.599372 22877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:22.599414 22877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:22.603565 22877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:22.603629 22877 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:10:22.603646 22877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:22.603665 22877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:22.603682 22877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:22.603742 22877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:22.609234 22877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:44707
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:22.610524 22877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:22.611558 22877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:22.620661 22885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:22.620661 22882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:22.620685 22883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:22.621186 22877 server_base.cc:1034] running on GCE node
I20260929 03:10:22.621786 22877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:22.623013 22877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:22.624209 22877 hybrid_clock.cc:648] HybridClock initialized: now 1790651422624196 us; error 33 us; skew 500 ppm
Sep 29 03:10:22 dist-test-slave-k1dg krb5kdc[22864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651422, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:22.627136 22877 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:22.628139 22877 webserver.cc:460] Webserver started at http://127.20.174.126:38841/ using document root <none> and password file <none>
I20260929 03:10:22.628719 22877 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:22.628813 22877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:22.629053 22877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:22.631246 22877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "035e76f333da44c1955acdcf683f8204"
format_stamp: "Formatted at 2026-09-29 03:10:22 on dist-test-slave-k1dg"
server_key: "45ee993a85bb0f985fb5271f0f8d2ebf"
server_key_iv: "a36b64dea77a50f5d29a34984eff3d4c"
server_key_version: "encryptionkey@0"
I20260929 03:10:22.631836 22877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "035e76f333da44c1955acdcf683f8204"
format_stamp: "Formatted at 2026-09-29 03:10:22 on dist-test-slave-k1dg"
server_key: "45ee993a85bb0f985fb5271f0f8d2ebf"
server_key_iv: "a36b64dea77a50f5d29a34984eff3d4c"
server_key_version: "encryptionkey@0"
I20260929 03:10:22.635797 22877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:10:22.638556 22892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:22.639860 22877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:22.640062 22877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "035e76f333da44c1955acdcf683f8204"
format_stamp: "Formatted at 2026-09-29 03:10:22 on dist-test-slave-k1dg"
server_key: "45ee993a85bb0f985fb5271f0f8d2ebf"
server_key_iv: "a36b64dea77a50f5d29a34984eff3d4c"
server_key_version: "encryptionkey@0"
I20260929 03:10:22.640198 22877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:22.661435 22877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:22.664999 22877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:22.665254 22877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:22.674237 22954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:44707 every 8 connection(s)
I20260929 03:10:22.674288 22877 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:44707
I20260929 03:10:22.675407 22877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:22.678700 22955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:22.680872 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 22877
I20260929 03:10:22.680985 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:22.681250 21177 external_mini_cluster.cc:1428] Setting key 6fc4b310af9125b2759f0d3525a70495
I20260929 03:10:22.685920 22955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204: Bootstrap starting.
I20260929 03:10:22.688755 22955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:22.689651 22955 log.cc:824] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:10:22 dist-test-slave-k1dg krb5kdc[22864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:22.692144 22955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204: No bootstrap required, opened a new log
I20260929 03:10:22.696859 22955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035e76f333da44c1955acdcf683f8204" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44707 } }
I20260929 03:10:22.697101 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:22.682715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.20.174.126:44707)
0929 03:10:22.683224 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:22.683239 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:22.683300 (+ 61us) server_negotiation.cc:407] Connection header received
0929 03:10:22.683944 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:22.683961 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:22.684302 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:22.684576 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:22.685509 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:22.686688 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:22.687927 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:22.688417 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:22.688539 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:22.691554 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:22.691560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:22.691578 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:22.691610 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:22.693701 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:22.694190 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:22.694196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:22.694202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:22.694348 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:22.694750 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:22.694754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:22.694756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:22.695286 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:22.695501 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:22.695746 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:10:22.696072 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":147,"threads_started":1}
I20260929 03:10:22.697170 22955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:22.697216 22955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035e76f333da44c1955acdcf683f8204, State: Initialized, Role: FOLLOWER
I20260929 03:10:22.697696 22955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035e76f333da44c1955acdcf683f8204" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44707 } }
I20260929 03:10:22.697861 22955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:22.697916 22955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:22.698043 22955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:22.699108 22955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035e76f333da44c1955acdcf683f8204" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44707 } }
I20260929 03:10:22.699525 22955 leader_election.cc:302] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 035e76f333da44c1955acdcf683f8204; no voters:
I20260929 03:10:22.699877 22955 leader_election.cc:288] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:22.700016 22960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:22.700280 22960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [term 1 LEADER]: Becoming Leader. State: Replica: 035e76f333da44c1955acdcf683f8204, State: Running, Role: LEADER
I20260929 03:10:22.700610 22960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035e76f333da44c1955acdcf683f8204" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44707 } }
I20260929 03:10:22.701223 22955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:22.702451 22961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "035e76f333da44c1955acdcf683f8204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035e76f333da44c1955acdcf683f8204" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44707 } } }
I20260929 03:10:22.702607 22961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:22.702888 22962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 035e76f333da44c1955acdcf683f8204. Latest consensus state: current_term: 1 leader_uuid: "035e76f333da44c1955acdcf683f8204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035e76f333da44c1955acdcf683f8204" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44707 } } }
I20260929 03:10:22.702998 22962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:22.703302 22969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:22.706256 22969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:22.712078 22969 catalog_manager.cc:1347] Generated new cluster ID: bf99b99c12f34c4686215ca39dcae34d
I20260929 03:10:22.712165 22969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:22.734368 22969 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:22.735988 22969 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:22.745909 22969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204: Generated new TSK 0
I20260929 03:10:22.746899 22969 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:10:22.782640 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44707
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:22.906929 22981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:22.907291 22981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:22.907393 22981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:22.912262 22981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:22.912370 22981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:22.912452 22981 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:10:22.912505 22981 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:10:22.912587 22981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:22.919716 22981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.174.126:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:22.920909 22981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:22.922024 22981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:22.932968 22986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:22.932977 22989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:22.933174 22981 server_base.cc:1034] running on GCE node
W20260929 03:10:22.933475 22987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:22.934001 22981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:22.934791 22981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:22.936035 22981 hybrid_clock.cc:648] HybridClock initialized: now 1790651422936017 us; error 36 us; skew 500 ppm
Sep 29 03:10:22 dist-test-slave-k1dg krb5kdc[22864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651422, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:22.939266 22981 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:10:22.940284 22981 webserver.cc:460] Webserver started at http://127.20.174.65:42795/ using document root <none> and password file <none>
I20260929 03:10:22.940900 22981 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:22.940979 22981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:22.941201 22981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:22.943447 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "fa6b479887b14e38a3f89582fcf16804"
format_stamp: "Formatted at 2026-09-29 03:10:22 on dist-test-slave-k1dg"
server_key: "3adf31f30e6b1d56b703c5d4a9b035ae"
server_key_iv: "b2d9b42eb12f9f27fbeab8faca932084"
server_key_version: "encryptionkey@0"
I20260929 03:10:22.944123 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "fa6b479887b14e38a3f89582fcf16804"
format_stamp: "Formatted at 2026-09-29 03:10:22 on dist-test-slave-k1dg"
server_key: "3adf31f30e6b1d56b703c5d4a9b035ae"
server_key_iv: "b2d9b42eb12f9f27fbeab8faca932084"
server_key_version: "encryptionkey@0"
I20260929 03:10:22.948503 22981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:22.951612 22996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:22.953226 22981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:10:22.953418 22981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "fa6b479887b14e38a3f89582fcf16804"
format_stamp: "Formatted at 2026-09-29 03:10:22 on dist-test-slave-k1dg"
server_key: "3adf31f30e6b1d56b703c5d4a9b035ae"
server_key_iv: "b2d9b42eb12f9f27fbeab8faca932084"
server_key_version: "encryptionkey@0"
I20260929 03:10:22.953564 22981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:22.972589 22981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:22.976706 22981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:22.976969 22981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:22.978837 22981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:22.978911 22981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:22.978968 22981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:22.978997 22981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:10:22 dist-test-slave-k1dg krb5kdc[22864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651422, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:22.991533 22981 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:42473
I20260929 03:10:22.991853 23111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:42473 every 8 connection(s)
I20260929 03:10:22.992628 22981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:22.994735 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:22.979703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:36667 (local address 127.20.174.126:44707)
0929 03:10:22.979845 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:22.979851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:22.981105 (+ 1254us) server_negotiation.cc:407] Connection header received
0929 03:10:22.982679 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:22.982684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:22.982774 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:22.982887 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:22.984540 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:22.985133 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:22.986525 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:22.986774 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:22.986907 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:22.990484 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:22.990489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:22.990492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:22.990525 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:22.992365 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:22.993016 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:22.993019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:22.993021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:22.993108 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:22.993478 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:22.993482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:22.993484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:22.993732 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:22.993830 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:22.994367 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 03:10:22.994558 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:10:22.995579 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:22.980336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44707 (local address 127.20.174.65:36667)
0929 03:10:22.980883 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:22.980955 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:22.982317 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:22.983154 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:22.983166 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:22.983666 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:22.984364 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:22.984383 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:22.985261 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:22.985265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:22.986342 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:22.986355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:22.986975 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:22.986980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:22.987231 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:22.987993 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:22.988011 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:22.990232 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:22.992519 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:22.992525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:22.992540 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:22.992886 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:22.993209 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:22.993213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:22.993215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:22.993363 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:22.993832 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:22.993840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:22.994105 (+ 265us) client_negotiation.cc:769] Sending connection context
0929 03:10:22.994303 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 03:10:22.994669 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":101,"threads_started":1}
I20260929 03:10:22.996955 23112 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44707
I20260929 03:10:22.997153 23112 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:22.997795 23112 heartbeater.cc:511] Master 127.20.174.126:44707 requested a full tablet report, sending...
I20260929 03:10:22.999569 22908 ts_manager.cc:194] Registered new tserver with Master: fa6b479887b14e38a3f89582fcf16804 (127.20.174.65:42473)
I20260929 03:10:23.000615 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 22981
I20260929 03:10:23.000741 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:23.001045 21177 external_mini_cluster.cc:1428] Setting key 10f51bd92441377c9d29effe839a1f84
I20260929 03:10:23.001178 22908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:36667
I20260929 03:10:23.008147 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:22.999356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44707 (local address 127.20.174.65:49275)
0929 03:10:22.999683 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:22.999698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:22.999841 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.000122 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.000125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.000382 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:23.000526 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.000534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.001590 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.001593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.002418 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.002431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.003096 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.003100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.003266 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.003854 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:23.003875 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:23.004234 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:23.006521 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.006525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.006526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.006743 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.007071 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.007073 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.007074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.007151 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.007650 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:23.007653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:23.007751 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.007837 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.007970 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":4,"thread_start_us":101,"threads_started":1}
I20260929 03:10:23.008306 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:22.999431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:49275 (local address 127.20.174.126:44707)
0929 03:10:22.999582 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:22.999586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:22.999768 (+ 182us) server_negotiation.cc:407] Connection header received
0929 03:10:22.999950 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:22.999952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.000000 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.000072 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:23.000671 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.001417 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.002595 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.002948 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.003078 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.004387 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:23.004391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:23.004394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:23.004421 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:23.006343 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.006855 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.006861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.006866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.006969 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.007250 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.007255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.007256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.007540 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:23.007658 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.007931 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.008092 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:10:23.034999 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44707
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:23.153224 23121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:23.153512 23121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:23.153563 23121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:23.158128 23121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:23.158201 23121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:23.158244 23121 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:10:23.158259 23121 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:10:23.158294 23121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:23.164083 23121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.174.126:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:23.165170 23121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:23.166181 23121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:23.177021 23129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:23.177045 23127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:23.177134 23121 server_base.cc:1034] running on GCE node
W20260929 03:10:23.177045 23126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:23.177803 23121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:23.178624 23121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:23.179845 23121 hybrid_clock.cc:648] HybridClock initialized: now 1790651423179813 us; error 43 us; skew 500 ppm
Sep 29 03:10:23 dist-test-slave-k1dg krb5kdc[22864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651423, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:23.183120 23121 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:10:23.184116 23121 webserver.cc:460] Webserver started at http://127.20.174.66:43517/ using document root <none> and password file <none>
I20260929 03:10:23.184715 23121 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:23.184819 23121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:23.185056 23121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:23.187319 23121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "04891df1b3d04ec7852856fb929e5f38"
format_stamp: "Formatted at 2026-09-29 03:10:23 on dist-test-slave-k1dg"
server_key: "75ecf80504a4d6f2571d9e342b249748"
server_key_iv: "49c8f5e3354de5c4234ee8ec06d660ef"
server_key_version: "encryptionkey@0"
I20260929 03:10:23.188024 23121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "04891df1b3d04ec7852856fb929e5f38"
format_stamp: "Formatted at 2026-09-29 03:10:23 on dist-test-slave-k1dg"
server_key: "75ecf80504a4d6f2571d9e342b249748"
server_key_iv: "49c8f5e3354de5c4234ee8ec06d660ef"
server_key_version: "encryptionkey@0"
I20260929 03:10:23.192119 23121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:23.194938 23136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:23.196345 23121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:23.196506 23121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "04891df1b3d04ec7852856fb929e5f38"
format_stamp: "Formatted at 2026-09-29 03:10:23 on dist-test-slave-k1dg"
server_key: "75ecf80504a4d6f2571d9e342b249748"
server_key_iv: "49c8f5e3354de5c4234ee8ec06d660ef"
server_key_version: "encryptionkey@0"
I20260929 03:10:23.196640 23121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:23.211675 23121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:23.215479 23121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:23.215704 23121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:23.217310 23121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:23.217406 23121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:23.217490 23121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:23.217525 23121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:10:23 dist-test-slave-k1dg krb5kdc[22864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651423, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:23.229562 23121 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:39879
I20260929 03:10:23.229682 23251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:39879 every 8 connection(s)
I20260929 03:10:23.230680 23121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:23.232038 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 23121
I20260929 03:10:23.232167 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:23.232447 21177 external_mini_cluster.cc:1428] Setting key 5fc6d22f2e8efcd87d37b41e010ebd62
I20260929 03:10:23.233124 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.218588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:54051 (local address 127.20.174.126:44707)
0929 03:10:23.218724 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.218728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.219979 (+ 1251us) server_negotiation.cc:407] Connection header received
0929 03:10:23.221383 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.221389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.221459 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.221562 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:23.223000 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.223670 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.224997 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.225295 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.225430 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.228777 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:23.228782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:23.228786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:23.228819 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:23.230571 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.231231 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.231235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.231237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.231334 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.231794 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.231800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.231802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.232074 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:23.232192 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.232829 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.232942 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:10:23.234131 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.219220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44707 (local address 127.20.174.66:54051)
0929 03:10:23.219768 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.219816 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.221138 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.221787 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.221806 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.222182 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:23.222839 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.222854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.223820 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.223823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.224864 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.224874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.225484 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.225489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.225675 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.226376 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:23.226399 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:23.228550 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:23.230694 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.230702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.230720 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.231095 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.231447 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.231452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.231454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.231621 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.232204 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:23.232212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:23.232507 (+ 295us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.232696 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.233062 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":133,"threads_started":1}
I20260929 03:10:23.235496 23252 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44707
I20260929 03:10:23.235677 23252 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:23.236193 23252 heartbeater.cc:511] Master 127.20.174.126:44707 requested a full tablet report, sending...
I20260929 03:10:23.238018 22908 ts_manager.cc:194] Registered new tserver with Master: 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879)
I20260929 03:10:23.238893 22908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:54051
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:10:23.245837 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.238074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:43089 (local address 127.20.174.126:44707)
0929 03:10:23.238205 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.238209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.238498 (+ 289us) server_negotiation.cc:407] Connection header received
0929 03:10:23.238626 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.238629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.238670 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.238736 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:23.239312 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.239890 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.240737 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.240968 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.241091 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.242257 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:23.242260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:23.242263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:23.242289 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:23.243909 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.244427 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.244431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.244433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.244529 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.244851 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.244855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.244857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.245162 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:23.245267 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.245544 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.245660 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:10:23.245846 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.238017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44707 (local address 127.20.174.66:43089)
0929 03:10:23.238379 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.238394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.238537 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.238785 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.238788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.239015 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:23.239148 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.239161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.240036 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.240039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.240605 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.240613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.241096 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.241100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.241191 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.241744 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:23.241756 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:23.242098 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:23.244099 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.244102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.244104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.244317 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.244662 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.244665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.244667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.244750 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.245255 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:23.245258 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:23.245362 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.245486 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.245648 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:10:23.266360 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44707
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:23.387639 23261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:23.387953 23261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:23.388011 23261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:23.392326 23261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:23.392398 23261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:23.392442 23261 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:10:23.392457 23261 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:10:23.392493 23261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:23.398316 23261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.174.126:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:23.399549 23261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:23.400712 23261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:23.411865 23266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:23.411885 23261 server_base.cc:1034] running on GCE node
W20260929 03:10:23.411828 23267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:23.411828 23269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:23.412557 23261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:23.413221 23261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:23.414419 23261 hybrid_clock.cc:648] HybridClock initialized: now 1790651423414402 us; error 39 us; skew 500 ppm
Sep 29 03:10:23 dist-test-slave-k1dg krb5kdc[22864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651423, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:23.417443 23261 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:10:23.418658 23261 webserver.cc:460] Webserver started at http://127.20.174.67:34217/ using document root <none> and password file <none>
I20260929 03:10:23.419277 23261 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:23.419380 23261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:23.419620 23261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:23.421844 23261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "e098266ba2af4254bb2fde59675ab3fe"
format_stamp: "Formatted at 2026-09-29 03:10:23 on dist-test-slave-k1dg"
server_key: "a493611c35cd2797e2a89dd887fdd34e"
server_key_iv: "96b12e36ee513b03a44dacced8ef2f37"
server_key_version: "encryptionkey@0"
I20260929 03:10:23.422523 23261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "e098266ba2af4254bb2fde59675ab3fe"
format_stamp: "Formatted at 2026-09-29 03:10:23 on dist-test-slave-k1dg"
server_key: "a493611c35cd2797e2a89dd887fdd34e"
server_key_iv: "96b12e36ee513b03a44dacced8ef2f37"
server_key_version: "encryptionkey@0"
I20260929 03:10:23.426743 23261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:10:23.429791 23276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:23.431492 23261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:23.431668 23261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "e098266ba2af4254bb2fde59675ab3fe"
format_stamp: "Formatted at 2026-09-29 03:10:23 on dist-test-slave-k1dg"
server_key: "a493611c35cd2797e2a89dd887fdd34e"
server_key_iv: "96b12e36ee513b03a44dacced8ef2f37"
server_key_version: "encryptionkey@0"
I20260929 03:10:23.431815 23261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:23.455729 23261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:23.459920 23261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:23.460165 23261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:23.461844 23261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:23.461967 23261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:23.462044 23261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:23.462098 23261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:10:23 dist-test-slave-k1dg krb5kdc[22864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651423, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:23.474740 23261 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:39595
I20260929 03:10:23.475574 23391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:39595 every 8 connection(s)
I20260929 03:10:23.475819 23261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:23.477777 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.463084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:40601 (local address 127.20.174.126:44707)
0929 03:10:23.463287 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.463292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.464324 (+ 1032us) server_negotiation.cc:407] Connection header received
0929 03:10:23.465787 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.465790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.465858 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.466044 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:23.468024 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.468812 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.470142 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.470398 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.470552 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.473610 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:23.473613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:23.473615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:23.473641 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:23.475371 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.476016 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.476020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.476022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.476134 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.476551 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.476555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.476556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.476799 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:23.476883 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.477541 (+ 658us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.477656 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 03:10:23.478868 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.463551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44707 (local address 127.20.174.67:40601)
0929 03:10:23.464107 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.464175 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.465502 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.466276 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.466291 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.466902 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:23.467813 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.467840 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.468967 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.468971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.469993 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.470005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.470725 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.470728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.470887 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.471553 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:23.471570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:23.473361 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:23.475513 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.475521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.475537 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.475880 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.476256 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.476260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.476261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.476432 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.476943 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:23.476950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:23.477279 (+ 329us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.477474 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.477767 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":101,"threads_started":1}
I20260929 03:10:23.480454 23392 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44707
I20260929 03:10:23.480657 23392 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:23.481194 23392 heartbeater.cc:511] Master 127.20.174.126:44707 requested a full tablet report, sending...
I20260929 03:10:23.482275 22908 ts_manager.cc:194] Registered new tserver with Master: e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595)
I20260929 03:10:23.483044 22908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:40601
I20260929 03:10:23.484068 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 23261
I20260929 03:10:23.484185 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:23.484510 21177 external_mini_cluster.cc:1428] Setting key 8eb94b361fe70dbdc882b7f2add7f964
I20260929 03:10:23.486675 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:10:23.491089 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.482751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44707 (local address 127.20.174.67:59951)
0929 03:10:23.483079 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.483095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.483205 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.483514 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.483517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.483778 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:23.483890 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.483896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.484755 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.484760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.485614 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.485626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.486143 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.486147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.486246 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.486892 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:23.486904 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:23.487255 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:23.489267 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.489274 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.489276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.489582 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.489870 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.489875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.489877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.489991 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.490617 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:23.490619 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:23.490704 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.490793 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.490923 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
I20260929 03:10:23.491292 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.482877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:59951 (local address 127.20.174.126:44707)
0929 03:10:23.483016 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.483021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.483211 (+ 190us) server_negotiation.cc:407] Connection header received
0929 03:10:23.483297 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.483301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.483394 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.483476 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:23.484067 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.484611 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.485740 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.486021 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.486104 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.487405 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:23.487412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:23.487418 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:23.487458 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:23.489130 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.489692 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.489695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.489697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.489775 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.490187 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.490194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.490198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.490511 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:23.490626 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.490884 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.491052 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 03:10:23 dist-test-slave-k1dg krb5kdc[22864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:10:23 dist-test-slave-k1dg krb5kdc[22864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:23.515472 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.506249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50006 (local address 127.20.174.126:44707)
0929 03:10:23.506423 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.506507 (+ 84us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.506520 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:10:23.506563 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.506566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.506604 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.506698 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:23.507500 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.508048 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.509098 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.509328 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.509457 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.512178 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:23.512181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:23.512193 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:23.512232 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:23.513676 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.514160 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.514163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.514165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.514241 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.514577 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.514581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.514582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.514793 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:23.514931 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.515233 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.515337 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:10:23.518697 22908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:23.521889 22908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:23.537669 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.531381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:39595 (local address 127.0.0.1:47074)
0929 03:10:23.531775 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.531834 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.531977 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.532761 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.532767 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.532792 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:23.532944 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.532953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.534315 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.534321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.535822 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.535834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.537055 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.537060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.537169 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.537212 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.537341 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.537510 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260929 03:10:23.537673 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.531528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.20.174.67:39595)
0929 03:10:23.531907 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.531923 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.531949 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:10:23.532164 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.532170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.532405 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.532565 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:23.533084 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.534205 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.535959 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.536930 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.537038 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.537207 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.537393 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.537499 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":132,"threads_started":1}
I20260929 03:10:23.537892 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.532021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:39879 (local address 127.0.0.1:35980)
0929 03:10:23.532705 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.532719 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.532804 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.533499 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.533502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.533519 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:23.533627 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.533637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.535072 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.535075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.536369 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.536379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.537515 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.537518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.537602 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.537622 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.537668 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.537747 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":25,"thread_start_us":102,"threads_started":1}
I20260929 03:10:23.538232 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.532170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:42473 (local address 127.0.0.1:44890)
0929 03:10:23.533070 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.533083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.533167 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.533787 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.533790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.533806 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:23.533892 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.533897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.535252 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.535257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.536490 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.536502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.537779 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.537783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.537898 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.537933 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.538032 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.538103 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260929 03:10:23.538297 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.532578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.20.174.65:42473)
0929 03:10:23.532935 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.532942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.533253 (+ 311us) server_negotiation.cc:407] Connection header received
0929 03:10:23.533341 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.533347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.533540 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.533666 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:23.534023 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.535121 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.536633 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.537468 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.537572 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.537710 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.538068 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.538167 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":66,"threads_started":1}
I20260929 03:10:23.538720 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.532339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.20.174.66:39879)
0929 03:10:23.533115 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.533130 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.533153 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:10:23.533220 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.533224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.533360 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.533523 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:23.533760 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.534951 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.536506 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.537385 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.537998 (+ 613us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.538182 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.538235 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.538332 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":104,"threads_started":1}
I20260929 03:10:23.541509 23186 tablet_service.cc:1467] Processing CreateTablet for tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf (DEFAULT_TABLE table=test-table [id=57bee8ed66e14f57bb76fa1a7b5a6bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:23.541512 23326 tablet_service.cc:1467] Processing CreateTablet for tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf (DEFAULT_TABLE table=test-table [id=57bee8ed66e14f57bb76fa1a7b5a6bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:23.541509 23046 tablet_service.cc:1467] Processing CreateTablet for tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf (DEFAULT_TABLE table=test-table [id=57bee8ed66e14f57bb76fa1a7b5a6bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:23.542599 23046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:23.542605 23326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:23.542620 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:23.550004 23415 tablet_bootstrap.cc:492] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: Bootstrap starting.
I20260929 03:10:23.550849 23417 tablet_bootstrap.cc:492] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: Bootstrap starting.
I20260929 03:10:23.553333 23417 tablet_bootstrap.cc:654] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:23.553709 23415 tablet_bootstrap.cc:654] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:23.554337 23417 log.cc:824] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:23.554750 23415 log.cc:824] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:23.555147 23416 tablet_bootstrap.cc:492] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: Bootstrap starting.
I20260929 03:10:23.556669 23417 tablet_bootstrap.cc:492] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: No bootstrap required, opened a new log
I20260929 03:10:23.556875 23417 ts_tablet_manager.cc:1397] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:10:23.557556 23416 tablet_bootstrap.cc:654] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:23.558455 23416 log.cc:824] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:23.559933 23415 tablet_bootstrap.cc:492] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: No bootstrap required, opened a new log
I20260929 03:10:23.560185 23415 ts_tablet_manager.cc:1397] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260929 03:10:23.560570 23416 tablet_bootstrap.cc:492] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: No bootstrap required, opened a new log
I20260929 03:10:23.560732 23416 ts_tablet_manager.cc:1397] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:10:23.561002 23417 raft_consensus.cc:348] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } }
I20260929 03:10:23.561283 23417 raft_consensus.cc:374] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:23.561334 23417 raft_consensus.cc:729] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e098266ba2af4254bb2fde59675ab3fe, State: Initialized, Role: FOLLOWER
I20260929 03:10:23.561867 23417 consensus_queue.cc:260] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } }
I20260929 03:10:23.562758 23392 heartbeater.cc:503] Master 127.20.174.126:44707 was elected leader, sending a full tablet report...
I20260929 03:10:23.563138 23417 ts_tablet_manager.cc:1428] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:10:23.564495 23415 raft_consensus.cc:348] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } }
I20260929 03:10:23.564769 23415 raft_consensus.cc:374] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:23.564826 23415 raft_consensus.cc:729] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04891df1b3d04ec7852856fb929e5f38, State: Initialized, Role: FOLLOWER
I20260929 03:10:23.565070 23416 raft_consensus.cc:348] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } }
I20260929 03:10:23.565304 23416 raft_consensus.cc:374] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:23.565371 23416 raft_consensus.cc:729] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6b479887b14e38a3f89582fcf16804, State: Initialized, Role: FOLLOWER
I20260929 03:10:23.565335 23415 consensus_queue.cc:260] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } }
I20260929 03:10:23.565793 23416 consensus_queue.cc:260] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } }
I20260929 03:10:23.566552 23112 heartbeater.cc:503] Master 127.20.174.126:44707 was elected leader, sending a full tablet report...
I20260929 03:10:23.566792 23252 heartbeater.cc:503] Master 127.20.174.126:44707 was elected leader, sending a full tablet report...
I20260929 03:10:23.566804 23416 ts_tablet_manager.cc:1428] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:10:23.567615 23415 ts_tablet_manager.cc:1428] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 03:10:23.673347 23421 raft_consensus.cc:482] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:23.673557 23421 raft_consensus.cc:504] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } }
I20260929 03:10:23.675048 23421 leader_election.cc:288] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6b479887b14e38a3f89582fcf16804 (127.20.174.65:42473), 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879)
I20260929 03:10:23.680891 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.675591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:33309 (local address 127.20.174.66:39879)
0929 03:10:23.675728 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.675732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.676166 (+ 434us) server_negotiation.cc:407] Connection header received
0929 03:10:23.676353 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.676357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.676408 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.676486 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:23.677212 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.677995 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.679487 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.680119 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.680211 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.680257 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.680667 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.680740 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:10:23.681105 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.675390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:42473 (local address 127.20.174.67:56929)
0929 03:10:23.675621 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.675636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.675751 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.676133 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.676146 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.676171 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:23.676312 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.676321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.677389 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.677395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.678912 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.678927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.680220 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.680226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.680385 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.680516 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.680656 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.680787 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":81}
I20260929 03:10:23.681574 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.675391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:39879 (local address 127.20.174.67:33309)
0929 03:10:23.676054 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.676094 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.676221 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.676492 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.676495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.676513 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:23.677063 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.677073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.678130 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.678133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.679339 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.679351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.680380 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.680384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.680540 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.680561 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.681316 (+ 755us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.681431 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":255,"thread_start_us":134,"threads_started":1}
I20260929 03:10:23.681928 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.675483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:56929 (local address 127.20.174.65:42473)
0929 03:10:23.675615 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.675620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.675766 (+ 146us) server_negotiation.cc:407] Connection header received
0929 03:10:23.675949 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.675952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.676002 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.676089 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:23.676452 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.677270 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.679105 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.679920 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.680065 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.680138 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.681555 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.681722 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:10:23.682703 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe0ecd4bbbc405ca9e6ab2a27f22dbf" candidate_uuid: "e098266ba2af4254bb2fde59675ab3fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04891df1b3d04ec7852856fb929e5f38" is_pre_election: true
I20260929 03:10:23.682909 23066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe0ecd4bbbc405ca9e6ab2a27f22dbf" candidate_uuid: "e098266ba2af4254bb2fde59675ab3fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6b479887b14e38a3f89582fcf16804" is_pre_election: true
I20260929 03:10:23.683095 23206 raft_consensus.cc:2454] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e098266ba2af4254bb2fde59675ab3fe in term 0.
I20260929 03:10:23.683193 23066 raft_consensus.cc:2454] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e098266ba2af4254bb2fde59675ab3fe in term 0.
I20260929 03:10:23.683719 23279 leader_election.cc:302] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04891df1b3d04ec7852856fb929e5f38, e098266ba2af4254bb2fde59675ab3fe, fa6b479887b14e38a3f89582fcf16804; no voters:
I20260929 03:10:23.684432 23421 raft_consensus.cc:2781] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:23.684544 23421 raft_consensus.cc:482] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:23.684593 23421 raft_consensus.cc:3037] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:23.685873 23421 raft_consensus.cc:504] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } }
I20260929 03:10:23.686249 23421 leader_election.cc:288] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [CANDIDATE]: Term 1 election: Requested vote from peers fa6b479887b14e38a3f89582fcf16804 (127.20.174.65:42473), 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879)
I20260929 03:10:23.686738 23066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe0ecd4bbbc405ca9e6ab2a27f22dbf" candidate_uuid: "e098266ba2af4254bb2fde59675ab3fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6b479887b14e38a3f89582fcf16804"
I20260929 03:10:23.686728 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe0ecd4bbbc405ca9e6ab2a27f22dbf" candidate_uuid: "e098266ba2af4254bb2fde59675ab3fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04891df1b3d04ec7852856fb929e5f38"
I20260929 03:10:23.686882 23206 raft_consensus.cc:3037] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:23.686882 23066 raft_consensus.cc:3037] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:23.688290 23206 raft_consensus.cc:2454] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e098266ba2af4254bb2fde59675ab3fe in term 1.
I20260929 03:10:23.688290 23066 raft_consensus.cc:2454] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e098266ba2af4254bb2fde59675ab3fe in term 1.
I20260929 03:10:23.688681 23279 leader_election.cc:302] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04891df1b3d04ec7852856fb929e5f38, e098266ba2af4254bb2fde59675ab3fe, fa6b479887b14e38a3f89582fcf16804; no voters:
I20260929 03:10:23.688860 23421 raft_consensus.cc:2781] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:23.689113 23421 raft_consensus.cc:686] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 1 LEADER]: Becoming Leader. State: Replica: e098266ba2af4254bb2fde59675ab3fe, State: Running, Role: LEADER
I20260929 03:10:23.689493 23421 consensus_queue.cc:237] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } }
I20260929 03:10:23.693291 22908 catalog_manager.cc:5495] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe reported cstate change: term changed from 0 to 1, leader changed from <none> to e098266ba2af4254bb2fde59675ab3fe (127.20.174.67). New cstate: current_term: 1 leader_uuid: "e098266ba2af4254bb2fde59675ab3fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } health_report { overall_health: UNKNOWN } } }
W20260929 03:10:23.726895 23393 tablet.cc:2369] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:10:23.731914 23253 tablet.cc:2369] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:10:23.744526 23113 tablet.cc:2369] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 03:10:23 dist-test-slave-k1dg krb5kdc[22864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651422, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:23.750049 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.740162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44707 (local address 127.0.0.1:50010)
0929 03:10:23.740494 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.740508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.740608 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.740891 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.740893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.741296 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:23.741392 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.741397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.742287 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.742290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.742967 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.742977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.743502 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.743509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.743612 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.744285 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:23.744300 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:23.746176 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:23.748216 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.748226 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.748231 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.748491 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.748788 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:23.748792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:23.748794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:23.748898 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:23.749382 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:23.749390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:23.749534 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.749660 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.749898 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260929 03:10:23.750133 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.740303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.20.174.126:44707)
0929 03:10:23.740554 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.740559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.740580 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:10:23.740716 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.740719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.740763 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.740846 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:23.741508 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.742166 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.743100 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.743315 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.743437 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.746506 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:23.746516 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:23.746519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:23.746557 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:23.748094 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.748590 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.748593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.748594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.748671 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:23.749011 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:23.749013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:23.749014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:23.749241 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:23.749324 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.749727 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.749982 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":123}
I20260929 03:10:23.754827 22908 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:50010:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:10:23.755560 22908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:23.760911 23186 tablet_service.cc:1467] Processing CreateTablet for tablet b1c9abf96c284207b57e769ef3b75025 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=583fb624a55a4315bc28690de527d2d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:10:23.760854 23046 tablet_service.cc:1467] Processing CreateTablet for tablet b1c9abf96c284207b57e769ef3b75025 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=583fb624a55a4315bc28690de527d2d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:10:23.761150 23326 tablet_service.cc:1467] Processing CreateTablet for tablet b1c9abf96c284207b57e769ef3b75025 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=583fb624a55a4315bc28690de527d2d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:10:23.761616 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c9abf96c284207b57e769ef3b75025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:23.761617 23046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c9abf96c284207b57e769ef3b75025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:23.761693 23326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c9abf96c284207b57e769ef3b75025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:23.764189 23415 tablet_bootstrap.cc:492] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38: Bootstrap starting.
I20260929 03:10:23.764675 23416 tablet_bootstrap.cc:492] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: Bootstrap starting.
I20260929 03:10:23.764889 23417 tablet_bootstrap.cc:492] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe: Bootstrap starting.
I20260929 03:10:23.765283 23415 tablet_bootstrap.cc:654] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:23.765698 23416 tablet_bootstrap.cc:654] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:23.765826 23417 tablet_bootstrap.cc:654] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:23.766758 23415 tablet_bootstrap.cc:492] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38: No bootstrap required, opened a new log
I20260929 03:10:23.766870 23415 ts_tablet_manager.cc:1397] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:23.767167 23416 tablet_bootstrap.cc:492] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: No bootstrap required, opened a new log
I20260929 03:10:23.767174 23417 tablet_bootstrap.cc:492] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe: No bootstrap required, opened a new log
I20260929 03:10:23.767258 23416 ts_tablet_manager.cc:1397] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:23.767258 23417 ts_tablet_manager.cc:1397] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:10:23.767632 23415 raft_consensus.cc:348] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.767710 23417 raft_consensus.cc:348] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.767710 23416 raft_consensus.cc:348] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.767802 23415 raft_consensus.cc:374] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:23.767838 23416 raft_consensus.cc:374] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:23.767838 23417 raft_consensus.cc:374] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:23.767855 23415 raft_consensus.cc:729] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04891df1b3d04ec7852856fb929e5f38, State: Initialized, Role: FOLLOWER
I20260929 03:10:23.767874 23416 raft_consensus.cc:729] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6b479887b14e38a3f89582fcf16804, State: Initialized, Role: FOLLOWER
I20260929 03:10:23.767874 23417 raft_consensus.cc:729] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e098266ba2af4254bb2fde59675ab3fe, State: Initialized, Role: FOLLOWER
I20260929 03:10:23.768016 23417 consensus_queue.cc:260] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.768016 23416 consensus_queue.cc:260] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.768172 23415 consensus_queue.cc:260] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.768283 23416 ts_tablet_manager.cc:1428] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:23.768289 23417 ts_tablet_manager.cc:1428] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:23.768453 23422 tablet_replica.cc:439] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } }
I20260929 03:10:23.768580 23422 tablet_replica.cc:442] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:23.768558 23421 tablet_replica.cc:439] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } }
I20260929 03:10:23.768680 23421 tablet_replica.cc:442] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:23.768867 23423 tablet_replica.cc:439] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } }
I20260929 03:10:23.769013 23423 tablet_replica.cc:442] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:23.769910 23415 ts_tablet_manager.cc:1428] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:23.784556 23423 raft_consensus.cc:482] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:23.784724 23423 raft_consensus.cc:504] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.785835 23423 leader_election.cc:288] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879), e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595)
I20260929 03:10:23.791103 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.786224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:56325 (local address 127.20.174.66:39879)
0929 03:10:23.786355 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.786360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.786722 (+ 362us) server_negotiation.cc:407] Connection header received
0929 03:10:23.786862 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.786864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.786904 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.787017 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:23.787374 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.788150 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.789641 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.790309 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.790496 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.790560 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.790877 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.790953 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:10:23.791152 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.786141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:39879 (local address 127.20.174.65:56325)
0929 03:10:23.786593 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.786607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.786743 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.787010 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.787013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.787031 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:23.787248 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.787255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.788270 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.788274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.789408 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.789418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.790500 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.790504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.790632 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.790654 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.790853 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.790943 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":84,"spinlock_wait_cycles":14208,"thread_start_us":133,"threads_started":1}
I20260929 03:10:23.791254 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.786223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:51781 (local address 127.20.174.67:39595)
0929 03:10:23.786355 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.786360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.786690 (+ 330us) server_negotiation.cc:407] Connection header received
0929 03:10:23.786860 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.786864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.786917 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.787021 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:23.788013 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.788800 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.790079 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.790622 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.790773 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.790827 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.791037 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.791114 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:10:23.791463 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.786141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:39595 (local address 127.20.174.65:51781)
0929 03:10:23.786578 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.786591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.786711 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.787048 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.787051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.787078 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:23.787833 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.787844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.789016 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.789019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.789936 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.789945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.790761 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.790765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.790864 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.790892 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.791015 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.791094 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":24,"thread_start_us":159,"threads_started":1}
I20260929 03:10:23.791572 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c9abf96c284207b57e769ef3b75025" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04891df1b3d04ec7852856fb929e5f38" is_pre_election: true
I20260929 03:10:23.791738 23206 raft_consensus.cc:2454] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 0.
I20260929 03:10:23.792200 22999 leader_election.cc:302] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04891df1b3d04ec7852856fb929e5f38, fa6b479887b14e38a3f89582fcf16804; no voters:
I20260929 03:10:23.792532 23423 raft_consensus.cc:2781] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:23.792613 23423 raft_consensus.cc:482] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:23.792524 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c9abf96c284207b57e769ef3b75025" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e098266ba2af4254bb2fde59675ab3fe" is_pre_election: true
I20260929 03:10:23.792661 23423 raft_consensus.cc:3037] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:23.792865 23346 raft_consensus.cc:2454] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 0.
I20260929 03:10:23.793944 23423 raft_consensus.cc:504] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.794299 23423 leader_election.cc:288] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 election: Requested vote from peers 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879), e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595)
I20260929 03:10:23.794706 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c9abf96c284207b57e769ef3b75025" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e098266ba2af4254bb2fde59675ab3fe"
I20260929 03:10:23.794736 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c9abf96c284207b57e769ef3b75025" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04891df1b3d04ec7852856fb929e5f38"
I20260929 03:10:23.794883 23346 raft_consensus.cc:3037] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:23.794902 23206 raft_consensus.cc:3037] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:23.795881 23206 raft_consensus.cc:2454] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 1.
I20260929 03:10:23.795900 23346 raft_consensus.cc:2454] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 1.
I20260929 03:10:23.796240 22999 leader_election.cc:302] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04891df1b3d04ec7852856fb929e5f38, e098266ba2af4254bb2fde59675ab3fe, fa6b479887b14e38a3f89582fcf16804; no voters:
I20260929 03:10:23.796429 23423 raft_consensus.cc:2781] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:23.796682 23423 raft_consensus.cc:686] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [term 1 LEADER]: Becoming Leader. State: Replica: fa6b479887b14e38a3f89582fcf16804, State: Running, Role: LEADER
I20260929 03:10:23.797081 23423 consensus_queue.cc:237] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.799531 23438 tablet_replica.cc:439] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: TxnStatusTablet state changed. Reason: New leader fa6b479887b14e38a3f89582fcf16804. Latest consensus state: current_term: 1 leader_uuid: "fa6b479887b14e38a3f89582fcf16804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } }
I20260929 03:10:23.799686 23438 tablet_replica.cc:442] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:10:23.799984 23440 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:10:23.800216 23440 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:10:23.801293 22909 catalog_manager.cc:5495] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6b479887b14e38a3f89582fcf16804 (127.20.174.65). New cstate: current_term: 1 leader_uuid: "fa6b479887b14e38a3f89582fcf16804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } health_report { overall_health: UNKNOWN } } }
I20260929 03:10:23.817059 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.812673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:42473 (local address 127.0.0.1:44900)
0929 03:10:23.812837 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.812854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.812961 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.813306 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.813309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.813324 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:10:23.813423 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.813428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.814569 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.814572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.815239 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.815248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.815822 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.815824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.815890 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.815988 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:10:23.816761 (+ 773us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:10:23.816776 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.816833 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.816906 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 03:10:23.817056 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.812822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.20.174.65:42473)
0929 03:10:23.812950 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.812955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.812970 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:23.813077 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.813080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.813146 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.813253 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:23.813541 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.814359 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.815361 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.815699 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.815814 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.816129 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:23.816653 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:23.816779 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.816855 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.816910 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:10:23.822805 23346 raft_consensus.cc:1261] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Refusing update from remote peer fa6b479887b14e38a3f89582fcf16804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:23.822818 23206 raft_consensus.cc:1261] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Refusing update from remote peer fa6b479887b14e38a3f89582fcf16804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:23.823549 23438 consensus_queue.cc:1035] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:23.823858 23423 consensus_queue.cc:1035] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:23.827744 23421 tablet_replica.cc:439] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe: TxnStatusTablet state changed. Reason: New leader fa6b479887b14e38a3f89582fcf16804. Latest consensus state: current_term: 1 leader_uuid: "fa6b479887b14e38a3f89582fcf16804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } }
I20260929 03:10:23.827894 23421 tablet_replica.cc:442] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:23.827904 23422 tablet_replica.cc:439] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38: TxnStatusTablet state changed. Reason: New leader fa6b479887b14e38a3f89582fcf16804. Latest consensus state: current_term: 1 leader_uuid: "fa6b479887b14e38a3f89582fcf16804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } }
I20260929 03:10:23.828032 23422 tablet_replica.cc:442] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:23.831764 23438 tablet_replica.cc:439] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa6b479887b14e38a3f89582fcf16804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } }
I20260929 03:10:23.831808 23423 tablet_replica.cc:439] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa6b479887b14e38a3f89582fcf16804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } }
I20260929 03:10:23.831908 23438 tablet_replica.cc:442] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:10:23.831924 23423 tablet_replica.cc:442] T b1c9abf96c284207b57e769ef3b75025 P fa6b479887b14e38a3f89582fcf16804: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:10:23.832000 23443 mvcc.cc:205] Tried to move back new op lower bound from 7334508231970742272 to 7334508231875919872. Current Snapshot: MvccSnapshot[applied={T|T < 7334508231970742272}]
I20260929 03:10:23.833359 23422 tablet_replica.cc:439] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa6b479887b14e38a3f89582fcf16804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } }
I20260929 03:10:23.833487 23422 tablet_replica.cc:442] T b1c9abf96c284207b57e769ef3b75025 P 04891df1b3d04ec7852856fb929e5f38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:23.837013 23421 tablet_replica.cc:439] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa6b479887b14e38a3f89582fcf16804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } } }
I20260929 03:10:23.837167 23421 tablet_replica.cc:442] T b1c9abf96c284207b57e769ef3b75025 P e098266ba2af4254bb2fde59675ab3fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:23.840306 23445 mvcc.cc:205] Tried to move back new op lower bound from 7334508231970742272 to 7334508231875919872. Current Snapshot: MvccSnapshot[applied={T|T < 7334508231970742272 or (T in {7334508231970742272})}]
I20260929 03:10:23.847061 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.842122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.20.174.67:39595)
0929 03:10:23.842243 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.842248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.842293 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:10:23.842513 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.842516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.842582 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.842700 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:23.842956 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.843758 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.845111 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.845465 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.845579 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.845914 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:23.846565 (+ 651us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:23.846684 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.846796 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.846875 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:10:23.859979 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.855997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:49103 (local address 127.20.174.65:42473)
0929 03:10:23.856139 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.856143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.856156 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:10:23.856277 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.856279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.856347 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.856420 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:23.856737 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.857348 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.858784 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.859026 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.859118 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.859525 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:23.859601 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:23.859662 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.859804 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.859863 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:10:23.860010 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.855787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:42473 (local address 127.20.174.67:49103)
0929 03:10:23.856058 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.856073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.856167 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.856459 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.856462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.856471 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:10:23.856591 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.856598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.857456 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.857460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.858629 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.858642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.859160 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.859164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.859267 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.859331 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:10:23.859697 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:10:23.859704 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.859772 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.859861 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260929 03:10:23.867538 23205 raft_consensus.cc:1261] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Refusing update from remote peer e098266ba2af4254bb2fde59675ab3fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:23.867799 23066 raft_consensus.cc:1261] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Refusing update from remote peer e098266ba2af4254bb2fde59675ab3fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:23.868222 23425 consensus_queue.cc:1035] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:23.868386 23421 consensus_queue.cc:1035] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:23.876843 23444 mvcc.cc:205] Tried to move back new op lower bound from 7334508232156487680 to 7334508231435182080. Current Snapshot: MvccSnapshot[applied={T|T < 7334508232156487680}]
I20260929 03:10:23.894977 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.890999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:36119 (local address 127.20.174.67:39595)
0929 03:10:23.891109 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:23.891113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:23.891128 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:23.891252 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:23.891255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:23.891304 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:23.891374 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:23.891654 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.892329 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.893605 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.893969 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.894104 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.894477 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:23.894568 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:23.894644 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:23.894757 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:10:23.894821 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:10:23.894994 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:23.890683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:39595 (local address 127.20.174.65:36119)
0929 03:10:23.891038 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:23.891054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:23.891159 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:23.891417 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:23.891420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:23.891428 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:10:23.891557 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.891563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.892457 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.892460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.893451 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:23.893462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:23.894116 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:23.894120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:23.894224 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:23.894292 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:10:23.894663 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:10:23.894670 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:10:23.894732 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:10:23.894818 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":127,"threads_started":1}
I20260929 03:10:23.921348 22909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:23.921557 22909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:23.924217 22909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204: Sending DeleteTablet for 3 replicas of tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf
I20260929 03:10:23.925055 23326 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:23 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:47074
I20260929 03:10:23.925272 23186 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:23 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:35980
I20260929 03:10:23.925770 23463 tablet_replica.cc:330] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: stopping tablet replica
I20260929 03:10:23.925843 23046 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:23 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:44890
I20260929 03:10:23.926240 23463 raft_consensus.cc:2229] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:23.926546 23463 raft_consensus.cc:2258] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:23.927138 22909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:23.927600 22909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:23.928501 23463 ts_tablet_manager.cc:1905] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:23.931988 23463 ts_tablet_manager.cc:1918] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:23.932200 23463 log.cc:1196] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/ebe0ecd4bbbc405ca9e6ab2a27f22dbf
I20260929 03:10:23.932646 23463 ts_tablet_manager.cc:1939] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P 04891df1b3d04ec7852856fb929e5f38: Deleting consensus metadata
I20260929 03:10:23.933352 23326 tablet_service.cc:1467] Processing CreateTablet for tablet 9d68229b803c400ea4504a17f1a1ab09 (DEFAULT_TABLE table=test-table [id=92cac079659d4090ab47695a4d615e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:23.933686 23326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d68229b803c400ea4504a17f1a1ab09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:23.934890 22895 catalog_manager.cc:4841] TS 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879): tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf (table test-table [id=57bee8ed66e14f57bb76fa1a7b5a6bb8]) successfully deleted
I20260929 03:10:23.936008 23464 tablet_replica.cc:330] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: stopping tablet replica
I20260929 03:10:23.936585 23186 tablet_service.cc:1467] Processing CreateTablet for tablet 9d68229b803c400ea4504a17f1a1ab09 (DEFAULT_TABLE table=test-table [id=92cac079659d4090ab47695a4d615e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:23.936762 23464 raft_consensus.cc:2229] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:23.936875 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d68229b803c400ea4504a17f1a1ab09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:23.937058 23464 raft_consensus.cc:2258] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:23.939543 23046 tablet_service.cc:1467] Processing CreateTablet for tablet 9d68229b803c400ea4504a17f1a1ab09 (DEFAULT_TABLE table=test-table [id=92cac079659d4090ab47695a4d615e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:23.939862 23046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d68229b803c400ea4504a17f1a1ab09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:23.941895 23416 tablet_bootstrap.cc:492] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804: Bootstrap starting.
I20260929 03:10:23.942975 23416 tablet_bootstrap.cc:654] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:23.943567 23417 tablet_bootstrap.cc:492] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe: Bootstrap starting.
I20260929 03:10:23.944505 23417 tablet_bootstrap.cc:654] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:23.946004 23417 tablet_bootstrap.cc:492] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe: No bootstrap required, opened a new log
I20260929 03:10:23.946130 23417 ts_tablet_manager.cc:1397] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:23.946638 23417 raft_consensus.cc:348] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.946775 23417 raft_consensus.cc:374] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:23.946815 23417 raft_consensus.cc:729] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e098266ba2af4254bb2fde59675ab3fe, State: Initialized, Role: FOLLOWER
I20260929 03:10:23.946933 23417 consensus_queue.cc:260] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.947212 23415 tablet_bootstrap.cc:492] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38: Bootstrap starting.
I20260929 03:10:23.947563 23462 tablet_replica.cc:330] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: stopping tablet replica
I20260929 03:10:23.948016 23462 raft_consensus.cc:2229] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:23.948341 23415 tablet_bootstrap.cc:654] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:23.950263 23464 ts_tablet_manager.cc:1905] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:23.950814 23462 raft_consensus.cc:2258] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:23.951192 23416 tablet_bootstrap.cc:492] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804: No bootstrap required, opened a new log
I20260929 03:10:23.951215 23417 ts_tablet_manager.cc:1428] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260929 03:10:23.951287 23416 ts_tablet_manager.cc:1397] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260929 03:10:23.951790 23416 raft_consensus.cc:348] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.951923 23416 raft_consensus.cc:374] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:23.951962 23416 raft_consensus.cc:729] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6b479887b14e38a3f89582fcf16804, State: Initialized, Role: FOLLOWER
I20260929 03:10:23.952085 23416 consensus_queue.cc:260] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.952381 23416 ts_tablet_manager.cc:1428] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:23.953094 23415 tablet_bootstrap.cc:492] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38: No bootstrap required, opened a new log
I20260929 03:10:23.953207 23415 ts_tablet_manager.cc:1397] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260929 03:10:23.953713 23415 raft_consensus.cc:348] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.953850 23415 raft_consensus.cc:374] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:23.953891 23415 raft_consensus.cc:729] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04891df1b3d04ec7852856fb929e5f38, State: Initialized, Role: FOLLOWER
I20260929 03:10:23.954059 23415 consensus_queue.cc:260] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.954342 23415 ts_tablet_manager.cc:1428] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:23.955693 23464 ts_tablet_manager.cc:1918] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:23.955695 23462 ts_tablet_manager.cc:1905] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:23.955809 23464 log.cc:1196] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/wal/wals/ebe0ecd4bbbc405ca9e6ab2a27f22dbf
I20260929 03:10:23.956147 23464 ts_tablet_manager.cc:1939] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P fa6b479887b14e38a3f89582fcf16804: Deleting consensus metadata
I20260929 03:10:23.957298 22894 catalog_manager.cc:4841] TS fa6b479887b14e38a3f89582fcf16804 (127.20.174.65:42473): tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf (table test-table [id=57bee8ed66e14f57bb76fa1a7b5a6bb8]) successfully deleted
I20260929 03:10:23.960391 23462 ts_tablet_manager.cc:1918] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:23.960536 23462 log.cc:1196] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/ebe0ecd4bbbc405ca9e6ab2a27f22dbf
I20260929 03:10:23.960899 23462 ts_tablet_manager.cc:1939] T ebe0ecd4bbbc405ca9e6ab2a27f22dbf P e098266ba2af4254bb2fde59675ab3fe: Deleting consensus metadata
I20260929 03:10:23.962143 22894 catalog_manager.cc:4841] TS e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595): tablet ebe0ecd4bbbc405ca9e6ab2a27f22dbf (table test-table [id=57bee8ed66e14f57bb76fa1a7b5a6bb8]) successfully deleted
I20260929 03:10:23.974086 23448 raft_consensus.cc:482] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:23.974225 23448 raft_consensus.cc:504] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.974761 23448 leader_election.cc:288] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879), e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595)
I20260929 03:10:23.975153 23205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68229b803c400ea4504a17f1a1ab09" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04891df1b3d04ec7852856fb929e5f38" is_pre_election: true
I20260929 03:10:23.975255 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68229b803c400ea4504a17f1a1ab09" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e098266ba2af4254bb2fde59675ab3fe" is_pre_election: true
I20260929 03:10:23.975303 23205 raft_consensus.cc:2454] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 0.
I20260929 03:10:23.975379 23346 raft_consensus.cc:2454] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 0.
I20260929 03:10:23.975657 22999 leader_election.cc:302] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04891df1b3d04ec7852856fb929e5f38, fa6b479887b14e38a3f89582fcf16804; no voters:
I20260929 03:10:23.975875 23448 raft_consensus.cc:2781] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:23.975947 23448 raft_consensus.cc:482] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:23.975996 23448 raft_consensus.cc:3037] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:23.977324 23448 raft_consensus.cc:504] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.977754 23448 leader_election.cc:288] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 election: Requested vote from peers 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879), e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595)
I20260929 03:10:23.978142 23205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68229b803c400ea4504a17f1a1ab09" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04891df1b3d04ec7852856fb929e5f38"
I20260929 03:10:23.978168 23346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68229b803c400ea4504a17f1a1ab09" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e098266ba2af4254bb2fde59675ab3fe"
I20260929 03:10:23.978273 23205 raft_consensus.cc:3037] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:23.978273 23346 raft_consensus.cc:3037] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:23.979494 23205 raft_consensus.cc:2454] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 1.
I20260929 03:10:23.979497 23346 raft_consensus.cc:2454] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 1.
I20260929 03:10:23.979859 22998 leader_election.cc:302] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e098266ba2af4254bb2fde59675ab3fe, fa6b479887b14e38a3f89582fcf16804; no voters:
I20260929 03:10:23.980062 23448 raft_consensus.cc:2781] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:23.980248 23448 raft_consensus.cc:686] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 1 LEADER]: Becoming Leader. State: Replica: fa6b479887b14e38a3f89582fcf16804, State: Running, Role: LEADER
I20260929 03:10:23.980427 23448 consensus_queue.cc:237] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:23.982297 22908 catalog_manager.cc:5495] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6b479887b14e38a3f89582fcf16804 (127.20.174.65). New cstate: current_term: 1 leader_uuid: "fa6b479887b14e38a3f89582fcf16804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } health_report { overall_health: UNKNOWN } } }
I20260929 03:10:24.006062 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:24.001400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.20.174.65:42473)
0929 03:10:24.001518 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:24.001523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:24.001535 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:10:24.001595 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:24.001598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:24.001648 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:24.001711 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:24.002175 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:24.003110 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:24.004477 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:24.004849 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:24.004944 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:24.005287 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:24.005372 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:24.005428 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:24.005856 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 03:10:24.005916 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:10:24.021370 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:24.017613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:42473 (local address 127.20.174.65:33919)
0929 03:10:24.017763 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:24.017778 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:24.017872 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:24.018125 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:24.018127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:24.018132 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:10:24.018235 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:24.018239 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:24.019284 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:24.019287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:24.019870 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:24.019880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:24.020479 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:24.020482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:24.020549 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:24.020579 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:10:24.021049 (+ 470us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:10:24.021056 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:10:24.021134 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:10:24.021239 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":2}
I20260929 03:10:24.021392 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:24.017717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:33919 (local address 127.20.174.65:42473)
0929 03:10:24.017854 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:24.017859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:24.017876 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:10:24.017952 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:24.017956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:24.018012 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:24.018109 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:24.018341 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:24.019153 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:24.019991 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:24.020334 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:24.020448 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:24.020702 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:24.020796 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:24.020869 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:24.021171 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 03:10:24.021254 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":65}
I20260929 03:10:24.028111 23205 raft_consensus.cc:1261] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Refusing update from remote peer fa6b479887b14e38a3f89582fcf16804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:24.028204 23346 raft_consensus.cc:1261] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Refusing update from remote peer fa6b479887b14e38a3f89582fcf16804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:24.028522 23423 consensus_queue.cc:1035] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:24.028651 23438 consensus_queue.cc:1035] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:10:24.045508 23443 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:10:25.607810 22908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:25.608057 22908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:25.610615 22908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204: Sending DeleteTablet for 3 replicas of tablet 9d68229b803c400ea4504a17f1a1ab09
I20260929 03:10:25.611222 23045 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d68229b803c400ea4504a17f1a1ab09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:25 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:44890
I20260929 03:10:25.611253 23326 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d68229b803c400ea4504a17f1a1ab09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:25 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:47074
I20260929 03:10:25.611569 23491 tablet_replica.cc:330] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804: stopping tablet replica
I20260929 03:10:25.611635 23492 tablet_replica.cc:330] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe: stopping tablet replica
I20260929 03:10:25.611711 23491 raft_consensus.cc:2229] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:25.611752 23492 raft_consensus.cc:2229] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:25.611871 23492 raft_consensus.cc:2258] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:25.612619 23492 ts_tablet_manager.cc:1905] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:25.612950 23491 raft_consensus.cc:2258] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:25.613420 23186 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d68229b803c400ea4504a17f1a1ab09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:25 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:35980
I20260929 03:10:25.613555 23491 ts_tablet_manager.cc:1905] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:25.613806 23494 tablet_replica.cc:330] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38: stopping tablet replica
I20260929 03:10:25.614010 23494 raft_consensus.cc:2229] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:25.614215 23494 raft_consensus.cc:2258] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:25.615103 23494 ts_tablet_manager.cc:1905] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:25.615742 23492 ts_tablet_manager.cc:1918] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:10:25.615824 23492 log.cc:1196] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/9d68229b803c400ea4504a17f1a1ab09
I20260929 03:10:25.616109 23492 ts_tablet_manager.cc:1939] T 9d68229b803c400ea4504a17f1a1ab09 P e098266ba2af4254bb2fde59675ab3fe: Deleting consensus metadata
I20260929 03:10:25.616626 23491 ts_tablet_manager.cc:1918] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:10:25.616708 23491 log.cc:1196] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/wal/wals/9d68229b803c400ea4504a17f1a1ab09
I20260929 03:10:25.616942 22894 catalog_manager.cc:4841] TS e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595): tablet 9d68229b803c400ea4504a17f1a1ab09 (table test-table [id=92cac079659d4090ab47695a4d615e0b]) successfully deleted
I20260929 03:10:25.617017 23491 ts_tablet_manager.cc:1939] T 9d68229b803c400ea4504a17f1a1ab09 P fa6b479887b14e38a3f89582fcf16804: Deleting consensus metadata
I20260929 03:10:25.617873 22894 catalog_manager.cc:4841] TS fa6b479887b14e38a3f89582fcf16804 (127.20.174.65:42473): tablet 9d68229b803c400ea4504a17f1a1ab09 (table test-table [id=92cac079659d4090ab47695a4d615e0b]) successfully deleted
I20260929 03:10:25.618412 23494 ts_tablet_manager.cc:1918] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:10:25.618530 23494 log.cc:1196] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/9d68229b803c400ea4504a17f1a1ab09
I20260929 03:10:25.618831 23494 ts_tablet_manager.cc:1939] T 9d68229b803c400ea4504a17f1a1ab09 P 04891df1b3d04ec7852856fb929e5f38: Deleting consensus metadata
I20260929 03:10:25.619650 22895 catalog_manager.cc:4841] TS 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879): tablet 9d68229b803c400ea4504a17f1a1ab09 (table test-table [id=92cac079659d4090ab47695a4d615e0b]) successfully deleted
Sep 29 03:10:25 dist-test-slave-k1dg krb5kdc[22864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651425, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:10:25 dist-test-slave-k1dg krb5kdc[22864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651425, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:25.642529 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:25.633164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.20.174.126:44707)
0929 03:10:25.633508 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:25.633519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:25.633536 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:10:25.633692 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:25.633696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:25.633749 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:25.633856 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:25.634641 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:25.635219 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:25.636200 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:25.636445 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:25.636561 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:25.639184 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:25.639188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:25.639195 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:25.639223 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:25.640669 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:25.641154 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:25.641157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:25.641159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:25.641248 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:25.641546 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:25.641550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:25.641552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:25.641749 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:25.641892 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:25.642145 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 03:10:25.642279 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":94,"threads_started":1}
I20260929 03:10:25.645532 22908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:25.646014 22908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:25.650995 23045 tablet_service.cc:1467] Processing CreateTablet for tablet 12a7428fb47d49779ec5dbfa95d5d1d8 (DEFAULT_TABLE table=test-table [id=f3c6c0f4668244968fdae6b76e56cce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:25.651274 23186 tablet_service.cc:1467] Processing CreateTablet for tablet 12a7428fb47d49779ec5dbfa95d5d1d8 (DEFAULT_TABLE table=test-table [id=f3c6c0f4668244968fdae6b76e56cce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:25.651274 23326 tablet_service.cc:1467] Processing CreateTablet for tablet 12a7428fb47d49779ec5dbfa95d5d1d8 (DEFAULT_TABLE table=test-table [id=f3c6c0f4668244968fdae6b76e56cce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:25.651355 23045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a7428fb47d49779ec5dbfa95d5d1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:25.651538 23326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a7428fb47d49779ec5dbfa95d5d1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:25.651537 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a7428fb47d49779ec5dbfa95d5d1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:25.653604 23502 tablet_bootstrap.cc:492] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804: Bootstrap starting.
I20260929 03:10:25.654027 23503 tablet_bootstrap.cc:492] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe: Bootstrap starting.
I20260929 03:10:25.654640 23504 tablet_bootstrap.cc:492] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38: Bootstrap starting.
I20260929 03:10:25.654724 23502 tablet_bootstrap.cc:654] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:25.654934 23503 tablet_bootstrap.cc:654] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:25.655655 23504 tablet_bootstrap.cc:654] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:25.656082 23502 tablet_bootstrap.cc:492] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804: No bootstrap required, opened a new log
I20260929 03:10:25.656215 23502 ts_tablet_manager.cc:1397] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:25.656337 23503 tablet_bootstrap.cc:492] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe: No bootstrap required, opened a new log
I20260929 03:10:25.656708 23502 raft_consensus.cc:348] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.656840 23502 raft_consensus.cc:374] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:25.656838 23503 ts_tablet_manager.cc:1397] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:25.656879 23502 raft_consensus.cc:729] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6b479887b14e38a3f89582fcf16804, State: Initialized, Role: FOLLOWER
I20260929 03:10:25.657012 23502 consensus_queue.cc:260] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.657285 23504 tablet_bootstrap.cc:492] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38: No bootstrap required, opened a new log
I20260929 03:10:25.657292 23502 ts_tablet_manager.cc:1428] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 03:10:25.657295 23503 raft_consensus.cc:348] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.657428 23503 raft_consensus.cc:374] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:25.657488 23503 raft_consensus.cc:729] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e098266ba2af4254bb2fde59675ab3fe, State: Initialized, Role: FOLLOWER
I20260929 03:10:25.657475 23504 ts_tablet_manager.cc:1397] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:25.657663 23503 consensus_queue.cc:260] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.658113 23503 ts_tablet_manager.cc:1428] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:25.658066 23504 raft_consensus.cc:348] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.658175 23504 raft_consensus.cc:374] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:25.658209 23504 raft_consensus.cc:729] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04891df1b3d04ec7852856fb929e5f38, State: Initialized, Role: FOLLOWER
I20260929 03:10:25.658327 23504 consensus_queue.cc:260] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.659010 23504 ts_tablet_manager.cc:1428] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:25.663223 23479 raft_consensus.cc:482] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:25.663349 23479 raft_consensus.cc:504] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.663800 23479 leader_election.cc:288] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879), e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595)
I20260929 03:10:25.664117 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a7428fb47d49779ec5dbfa95d5d1d8" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04891df1b3d04ec7852856fb929e5f38" is_pre_election: true
I20260929 03:10:25.664160 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a7428fb47d49779ec5dbfa95d5d1d8" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e098266ba2af4254bb2fde59675ab3fe" is_pre_election: true
I20260929 03:10:25.664271 23345 raft_consensus.cc:2454] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 0.
I20260929 03:10:25.664268 23206 raft_consensus.cc:2454] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 0.
I20260929 03:10:25.664593 22998 leader_election.cc:302] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e098266ba2af4254bb2fde59675ab3fe, fa6b479887b14e38a3f89582fcf16804; no voters:
I20260929 03:10:25.664781 23479 raft_consensus.cc:2781] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:25.664847 23479 raft_consensus.cc:482] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:25.664875 23479 raft_consensus.cc:3037] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:25.665907 23479 raft_consensus.cc:504] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.666241 23479 leader_election.cc:288] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 election: Requested vote from peers 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879), e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595)
I20260929 03:10:25.666544 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a7428fb47d49779ec5dbfa95d5d1d8" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04891df1b3d04ec7852856fb929e5f38"
I20260929 03:10:25.666718 23206 raft_consensus.cc:3037] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:25.666714 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a7428fb47d49779ec5dbfa95d5d1d8" candidate_uuid: "fa6b479887b14e38a3f89582fcf16804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e098266ba2af4254bb2fde59675ab3fe"
I20260929 03:10:25.666838 23345 raft_consensus.cc:3037] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:25.668016 23206 raft_consensus.cc:2454] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 1.
I20260929 03:10:25.668128 23345 raft_consensus.cc:2454] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6b479887b14e38a3f89582fcf16804 in term 1.
I20260929 03:10:25.668408 22998 leader_election.cc:302] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04891df1b3d04ec7852856fb929e5f38, e098266ba2af4254bb2fde59675ab3fe, fa6b479887b14e38a3f89582fcf16804; no voters:
I20260929 03:10:25.668669 23479 raft_consensus.cc:2781] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:25.668742 23479 raft_consensus.cc:686] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 1 LEADER]: Becoming Leader. State: Replica: fa6b479887b14e38a3f89582fcf16804, State: Running, Role: LEADER
I20260929 03:10:25.668915 23479 consensus_queue.cc:237] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.670614 22908 catalog_manager.cc:5495] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6b479887b14e38a3f89582fcf16804 (127.20.174.65). New cstate: current_term: 1 leader_uuid: "fa6b479887b14e38a3f89582fcf16804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } health_report { overall_health: UNKNOWN } } }
I20260929 03:10:25.692746 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:25.689375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.20.174.65:42473)
0929 03:10:25.689663 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:25.689672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:25.689684 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:10:25.689734 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:25.689737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:25.689780 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:25.689875 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:25.690120 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:25.690751 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:25.691636 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:25.691885 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:25.692002 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:25.692273 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:25.692355 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:25.692436 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:25.692535 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:10:25.692588 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
I20260929 03:10:25.699587 23206 raft_consensus.cc:1261] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Refusing update from remote peer fa6b479887b14e38a3f89582fcf16804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:25.699888 23345 raft_consensus.cc:1261] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Refusing update from remote peer fa6b479887b14e38a3f89582fcf16804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:25.700137 23479 consensus_queue.cc:1035] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:25.700541 23474 consensus_queue.cc:1035] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:25.735440 22909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:25.735647 22909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:25.737905 22909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204: Sending DeleteTablet for 3 replicas of tablet 12a7428fb47d49779ec5dbfa95d5d1d8
I20260929 03:10:25.738655 23046 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a7428fb47d49779ec5dbfa95d5d1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:25 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:44890
I20260929 03:10:25.738665 23326 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a7428fb47d49779ec5dbfa95d5d1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:25 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:47074
I20260929 03:10:25.738734 23186 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a7428fb47d49779ec5dbfa95d5d1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:25 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:35980
I20260929 03:10:25.738914 23491 tablet_replica.cc:330] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804: stopping tablet replica
I20260929 03:10:25.738934 23494 tablet_replica.cc:330] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38: stopping tablet replica
I20260929 03:10:25.739044 23494 raft_consensus.cc:2229] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:25.739044 23491 raft_consensus.cc:2229] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:25.739172 23494 raft_consensus.cc:2258] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:25.739265 23491 raft_consensus.cc:2258] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:25.739475 23492 tablet_replica.cc:330] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe: stopping tablet replica
I20260929 03:10:25.739622 23492 raft_consensus.cc:2229] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:25.739776 23492 raft_consensus.cc:2258] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:25.739732 22909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:25.740191 22909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:25.740584 23494 ts_tablet_manager.cc:1905] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:25.741389 23491 ts_tablet_manager.cc:1905] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:25.743598 23492 ts_tablet_manager.cc:1905] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:25.744352 23491 ts_tablet_manager.cc:1918] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:25.744438 23491 log.cc:1196] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-0/wal/wals/12a7428fb47d49779ec5dbfa95d5d1d8
I20260929 03:10:25.744711 23491 ts_tablet_manager.cc:1939] T 12a7428fb47d49779ec5dbfa95d5d1d8 P fa6b479887b14e38a3f89582fcf16804: Deleting consensus metadata
I20260929 03:10:25.745748 22894 catalog_manager.cc:4841] TS fa6b479887b14e38a3f89582fcf16804 (127.20.174.65:42473): tablet 12a7428fb47d49779ec5dbfa95d5d1d8 (table test-table [id=f3c6c0f4668244968fdae6b76e56cce9]) successfully deleted
I20260929 03:10:25.745918 23326 tablet_service.cc:1467] Processing CreateTablet for tablet c737d72bb2424d768e925a6967078f54 (DEFAULT_TABLE table=test-table [id=b83428c52b414206aedf004edcd6cb4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:25.746234 23326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c737d72bb2424d768e925a6967078f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:25.746374 23492 ts_tablet_manager.cc:1918] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:25.746474 23492 log.cc:1196] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/12a7428fb47d49779ec5dbfa95d5d1d8
I20260929 03:10:25.746733 23492 ts_tablet_manager.cc:1939] T 12a7428fb47d49779ec5dbfa95d5d1d8 P e098266ba2af4254bb2fde59675ab3fe: Deleting consensus metadata
I20260929 03:10:25.746716 23186 tablet_service.cc:1467] Processing CreateTablet for tablet c737d72bb2424d768e925a6967078f54 (DEFAULT_TABLE table=test-table [id=b83428c52b414206aedf004edcd6cb4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:25.747143 23046 tablet_service.cc:1467] Processing CreateTablet for tablet c737d72bb2424d768e925a6967078f54 (DEFAULT_TABLE table=test-table [id=b83428c52b414206aedf004edcd6cb4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:25.747406 23046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c737d72bb2424d768e925a6967078f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:25.747457 22894 catalog_manager.cc:4841] TS e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595): tablet 12a7428fb47d49779ec5dbfa95d5d1d8 (table test-table [id=f3c6c0f4668244968fdae6b76e56cce9]) successfully deleted
I20260929 03:10:25.748662 23503 tablet_bootstrap.cc:492] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe: Bootstrap starting.
I20260929 03:10:25.749378 23503 tablet_bootstrap.cc:654] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:25.749632 23502 tablet_bootstrap.cc:492] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804: Bootstrap starting.
I20260929 03:10:25.749778 23494 ts_tablet_manager.cc:1918] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:25.749876 23494 log.cc:1196] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/12a7428fb47d49779ec5dbfa95d5d1d8
I20260929 03:10:25.750154 23494 ts_tablet_manager.cc:1939] T 12a7428fb47d49779ec5dbfa95d5d1d8 P 04891df1b3d04ec7852856fb929e5f38: Deleting consensus metadata
I20260929 03:10:25.750411 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c737d72bb2424d768e925a6967078f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:25.750495 23502 tablet_bootstrap.cc:654] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:25.750602 23503 tablet_bootstrap.cc:492] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe: No bootstrap required, opened a new log
I20260929 03:10:25.750674 23503 ts_tablet_manager.cc:1397] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:10:25.751062 23503 raft_consensus.cc:348] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.751147 23503 raft_consensus.cc:374] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:25.751168 23503 raft_consensus.cc:729] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e098266ba2af4254bb2fde59675ab3fe, State: Initialized, Role: FOLLOWER
I20260929 03:10:25.751304 23503 consensus_queue.cc:260] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.751579 23503 ts_tablet_manager.cc:1428] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:25.751744 22895 catalog_manager.cc:4841] TS 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879): tablet 12a7428fb47d49779ec5dbfa95d5d1d8 (table test-table [id=f3c6c0f4668244968fdae6b76e56cce9]) successfully deleted
I20260929 03:10:25.752770 23504 tablet_bootstrap.cc:492] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38: Bootstrap starting.
I20260929 03:10:25.753499 23502 tablet_bootstrap.cc:492] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804: No bootstrap required, opened a new log
I20260929 03:10:25.753580 23502 ts_tablet_manager.cc:1397] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 03:10:25.753778 23504 tablet_bootstrap.cc:654] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:25.754091 23502 raft_consensus.cc:348] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.754221 23502 raft_consensus.cc:374] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:25.754259 23502 raft_consensus.cc:729] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6b479887b14e38a3f89582fcf16804, State: Initialized, Role: FOLLOWER
I20260929 03:10:25.754400 23502 consensus_queue.cc:260] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.754756 23502 ts_tablet_manager.cc:1428] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 03:10:25.755286 23504 tablet_bootstrap.cc:492] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38: No bootstrap required, opened a new log
I20260929 03:10:25.755406 23504 ts_tablet_manager.cc:1397] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:25.755807 23504 raft_consensus.cc:348] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.755934 23504 raft_consensus.cc:374] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:25.755980 23504 raft_consensus.cc:729] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04891df1b3d04ec7852856fb929e5f38, State: Initialized, Role: FOLLOWER
I20260929 03:10:25.756108 23504 consensus_queue.cc:260] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:25.756433 23504 ts_tablet_manager.cc:1428] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:10:26.053531 23509 raft_consensus.cc:482] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:26.053702 23509 raft_consensus.cc:504] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:26.055044 23509 leader_election.cc:288] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6b479887b14e38a3f89582fcf16804 (127.20.174.65:42473), e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595)
I20260929 03:10:26.060014 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:26.055311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:42473 (local address 127.20.174.66:40519)
0929 03:10:26.055603 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:26.055622 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:26.055756 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:26.056125 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:26.056129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:26.056152 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:26.056322 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:26.056330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.057364 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.057370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:26.058732 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:26.058743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.059588 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.059591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:26.059681 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:26.059703 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:10:26.059786 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:10:26.059855 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":125,"threads_started":1}
I20260929 03:10:26.060060 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:26.055561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:40519 (local address 127.20.174.65:42473)
0929 03:10:26.055731 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:26.055736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:26.055753 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:10:26.055919 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:26.055923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:26.055975 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:26.056079 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:26.056494 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.057256 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.058883 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.059379 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.059578 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:26.059638 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:26.059823 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 03:10:26.059905 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:10:26.060757 23066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c737d72bb2424d768e925a6967078f54" candidate_uuid: "04891df1b3d04ec7852856fb929e5f38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6b479887b14e38a3f89582fcf16804" is_pre_election: true
I20260929 03:10:26.060849 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:26.055940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:43981 (local address 127.20.174.67:39595)
0929 03:10:26.056207 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:26.056223 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:26.056393 (+ 170us) server_negotiation.cc:407] Connection header received
0929 03:10:26.056527 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:26.056531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:26.056586 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:26.056659 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:26.057046 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.057961 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.059371 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.060055 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.060199 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:26.060263 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:26.060654 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 03:10:26.060743 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260929 03:10:26.060909 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:26.055761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:39595 (local address 127.20.174.66:43981)
0929 03:10:26.056287 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:26.056299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:26.056390 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:26.056696 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:26.056698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:26.056717 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:26.056847 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:26.056853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.058094 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.058098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:26.059232 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:26.059240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.060363 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.060367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:26.060501 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:26.060525 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:10:26.060612 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:10:26.060731 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":65,"thread_start_us":131,"threads_started":1}
I20260929 03:10:26.060941 23066 raft_consensus.cc:2454] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04891df1b3d04ec7852856fb929e5f38 in term 0.
I20260929 03:10:26.061410 23138 leader_election.cc:302] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04891df1b3d04ec7852856fb929e5f38, fa6b479887b14e38a3f89582fcf16804; no voters:
I20260929 03:10:26.061429 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c737d72bb2424d768e925a6967078f54" candidate_uuid: "04891df1b3d04ec7852856fb929e5f38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e098266ba2af4254bb2fde59675ab3fe" is_pre_election: true
I20260929 03:10:26.061604 23345 raft_consensus.cc:2454] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04891df1b3d04ec7852856fb929e5f38 in term 0.
I20260929 03:10:26.061729 23509 raft_consensus.cc:2781] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:26.061787 23509 raft_consensus.cc:482] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:26.061826 23509 raft_consensus.cc:3037] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:26.063256 23509 raft_consensus.cc:504] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:26.063644 23509 leader_election.cc:288] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [CANDIDATE]: Term 1 election: Requested vote from peers fa6b479887b14e38a3f89582fcf16804 (127.20.174.65:42473), e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595)
I20260929 03:10:26.064078 23066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c737d72bb2424d768e925a6967078f54" candidate_uuid: "04891df1b3d04ec7852856fb929e5f38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6b479887b14e38a3f89582fcf16804"
I20260929 03:10:26.064146 23345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c737d72bb2424d768e925a6967078f54" candidate_uuid: "04891df1b3d04ec7852856fb929e5f38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e098266ba2af4254bb2fde59675ab3fe"
I20260929 03:10:26.064244 23066 raft_consensus.cc:3037] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:26.064278 23345 raft_consensus.cc:3037] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:26.065385 23345 raft_consensus.cc:2454] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04891df1b3d04ec7852856fb929e5f38 in term 1.
I20260929 03:10:26.065392 23066 raft_consensus.cc:2454] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04891df1b3d04ec7852856fb929e5f38 in term 1.
I20260929 03:10:26.065721 23138 leader_election.cc:302] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04891df1b3d04ec7852856fb929e5f38, fa6b479887b14e38a3f89582fcf16804; no voters:
I20260929 03:10:26.065907 23509 raft_consensus.cc:2781] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:26.066136 23509 raft_consensus.cc:686] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 1 LEADER]: Becoming Leader. State: Replica: 04891df1b3d04ec7852856fb929e5f38, State: Running, Role: LEADER
I20260929 03:10:26.066565 23509 consensus_queue.cc:237] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } }
I20260929 03:10:26.069468 22907 catalog_manager.cc:5495] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66). New cstate: current_term: 1 leader_uuid: "04891df1b3d04ec7852856fb929e5f38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04891df1b3d04ec7852856fb929e5f38" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 } health_report { overall_health: UNKNOWN } } }
I20260929 03:10:26.128158 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:26.123967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.20.174.66:39879)
0929 03:10:26.124382 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:26.124391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:26.124403 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:10:26.124491 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:26.124493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:26.124546 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:26.124683 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:26.124956 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.125592 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.126490 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.126770 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.126893 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:26.127202 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:26.127688 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:26.127849 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:26.127912 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 03:10:26.127978 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":180,"threads_started":1}
I20260929 03:10:26.140055 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:26.136372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:60117 (local address 127.20.174.65:42473)
0929 03:10:26.136584 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:26.136590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:26.136844 (+ 254us) server_negotiation.cc:407] Connection header received
0929 03:10:26.136972 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:26.136975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:26.137029 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:26.137100 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:26.137364 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.137887 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.138972 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.139199 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.139289 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:26.139627 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:26.139710 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:26.139767 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:26.139878 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 03:10:26.139929 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 03:10:26.140097 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:26.136279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:42473 (local address 127.20.174.66:60117)
0929 03:10:26.136747 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:26.136761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:26.136868 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:26.137134 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:26.137137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:26.137144 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:10:26.137259 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:26.137264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.138011 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.138014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:26.138831 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:26.138842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.139320 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.139323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:26.139404 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:26.139485 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:10:26.139794 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:10:26.139800 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:10:26.139856 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:10:26.139933 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":185,"threads_started":1}
I20260929 03:10:26.147037 23345 raft_consensus.cc:1261] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Refusing update from remote peer 04891df1b3d04ec7852856fb929e5f38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:26.147097 23066 raft_consensus.cc:1261] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Refusing update from remote peer 04891df1b3d04ec7852856fb929e5f38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:26.147703 23509 consensus_queue.cc:1035] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e098266ba2af4254bb2fde59675ab3fe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:26.147957 23522 consensus_queue.cc:1035] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6b479887b14e38a3f89582fcf16804" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:26.166782 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:26.163283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:50979 (local address 127.20.174.66:39879)
0929 03:10:26.163414 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:26.163419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:26.163433 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:10:26.163525 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:26.163528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:26.163574 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:26.163640 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:26.163925 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.164570 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.165513 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.165832 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.165966 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:26.166246 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:26.166351 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:26.166422 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:26.166575 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:10:26.166640 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:10:26.166829 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:26.163036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:39879 (local address 127.20.174.65:50979)
0929 03:10:26.163295 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:26.163311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:26.163413 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:26.163672 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:26.163677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:26.163689 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:10:26.163803 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:26.163808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.164690 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.164696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:26.165343 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:26.165351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:26.165967 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:26.165971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:26.166062 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:26.166099 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:10:26.166422 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:10:26.166470 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 03:10:26.166549 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:10:26.166659 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":114,"threads_started":1}
W20260929 03:10:26.167626 23475 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:10:27.736209 22907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:27.736392 22907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:27.739260 22907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 035e76f333da44c1955acdcf683f8204: Sending DeleteTablet for 3 replicas of tablet c737d72bb2424d768e925a6967078f54
I20260929 03:10:27.740028 23186 tablet_service.cc:1514] Processing DeleteTablet for tablet c737d72bb2424d768e925a6967078f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:27 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:35980
I20260929 03:10:27.740123 23326 tablet_service.cc:1514] Processing DeleteTablet for tablet c737d72bb2424d768e925a6967078f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:27 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:47074
I20260929 03:10:27.740192 23046 tablet_service.cc:1514] Processing DeleteTablet for tablet c737d72bb2424d768e925a6967078f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:27 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:44890
I20260929 03:10:27.740556 23557 tablet_replica.cc:330] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804: stopping tablet replica
I20260929 03:10:27.740702 23557 raft_consensus.cc:2229] T c737d72bb2424d768e925a6967078f54 P fa6b479887b14e38a3f89582fcf16804 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:27.740844 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 22981
I20260929 03:10:27.742377 23558 tablet_replica.cc:330] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe: stopping tablet replica
I20260929 03:10:27.742573 23558 raft_consensus.cc:2229] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:27.742739 23558 raft_consensus.cc:2258] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:27.742847 23556 tablet_replica.cc:330] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38: stopping tablet replica
I20260929 03:10:27.743047 23556 raft_consensus.cc:2229] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:27.743485 23556 raft_consensus.cc:2258] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:27.743592 23558 ts_tablet_manager.cc:1905] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:27.744125 23556 ts_tablet_manager.cc:1905] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:27.746770 23558 ts_tablet_manager.cc:1918] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:10:27.746770 23556 ts_tablet_manager.cc:1918] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:10:27.746860 23556 log.cc:1196] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/c737d72bb2424d768e925a6967078f54
I20260929 03:10:27.746876 23558 log.cc:1196] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/c737d72bb2424d768e925a6967078f54
I20260929 03:10:27.747159 23558 ts_tablet_manager.cc:1939] T c737d72bb2424d768e925a6967078f54 P e098266ba2af4254bb2fde59675ab3fe: Deleting consensus metadata
I20260929 03:10:27.747166 23556 ts_tablet_manager.cc:1939] T c737d72bb2424d768e925a6967078f54 P 04891df1b3d04ec7852856fb929e5f38: Deleting consensus metadata
I20260929 03:10:27.748450 22894 catalog_manager.cc:4841] TS e098266ba2af4254bb2fde59675ab3fe (127.20.174.67:39595): tablet c737d72bb2424d768e925a6967078f54 (table test-table [id=b83428c52b414206aedf004edcd6cb4c]) successfully deleted
I20260929 03:10:27.748692 22895 catalog_manager.cc:4841] TS 04891df1b3d04ec7852856fb929e5f38 (127.20.174.66:39879): tablet c737d72bb2424d768e925a6967078f54 (table test-table [id=b83428c52b414206aedf004edcd6cb4c]) successfully deleted
W20260929 03:10:27.749593 22894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:42473 (error 108)
I20260929 03:10:27.750481 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 23121
I20260929 03:10:27.751863 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:27.750479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:42473 (local address 127.0.0.1:44940)
0929 03:10:27.751695 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:27.751769 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":85,"threads_started":1}
W20260929 03:10:27.752069 22894 catalog_manager.cc:4584] TS fa6b479887b14e38a3f89582fcf16804 (127.20.174.65:42473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c737d72bb2424d768e925a6967078f54: Network error: Client connection negotiation failed: client connection to 127.20.174.65:42473: connect: Connection refused (error 111)
I20260929 03:10:27.758157 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 23261
I20260929 03:10:27.765075 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 22877
2026-09-29T03:10:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6476 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:27 dist-test-slave-k1dg krb5kdc[23563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:27 dist-test-slave-k1dg krb5kdc[23563](info): set up 2 sockets
Sep 29 03:10:27 dist-test-slave-k1dg krb5kdc[23563](info): commencing operation
krb5kdc: starting...
W20260929 03:10:28.838701 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:28 dist-test-slave-k1dg krb5kdc[23563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:28.974231 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:36379
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:36379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:29.092602 23576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:29.092974 23576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:29.093062 23576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:29.097249 23576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:29.097370 23576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:29.097410 23576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:29.097450 23576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:29.097486 23576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:29.103662 23576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:36379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:29.104862 23576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:29.105957 23576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:29.114816 23581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:29.114817 23584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:29.114851 23582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:29.114880 23576 server_base.cc:1034] running on GCE node
I20260929 03:10:29.115628 23576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:29.116825 23576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:29.118012 23576 hybrid_clock.cc:648] HybridClock initialized: now 1790651429117999 us; error 28 us; skew 500 ppm
Sep 29 03:10:29 dist-test-slave-k1dg krb5kdc[23563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:29.121085 23576 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:29.122104 23576 webserver.cc:460] Webserver started at http://127.20.174.126:42389/ using document root <none> and password file <none>
I20260929 03:10:29.122771 23576 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:29.122840 23576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:29.123080 23576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:29.125262 23576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "0f10f10948be44278c6baf6fb8ce1475"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "09180ff16e8cbf550aa05c36f090b5c0"
server_key_iv: "ccb95b8d9dd07bc1bc435f2b393553f0"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.125885 23576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "0f10f10948be44278c6baf6fb8ce1475"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "09180ff16e8cbf550aa05c36f090b5c0"
server_key_iv: "ccb95b8d9dd07bc1bc435f2b393553f0"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.130204 23576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 03:10:29.133100 23591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:29.134645 23576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:29.134805 23576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "0f10f10948be44278c6baf6fb8ce1475"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "09180ff16e8cbf550aa05c36f090b5c0"
server_key_iv: "ccb95b8d9dd07bc1bc435f2b393553f0"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.134943 23576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:29.154410 23576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:29.157958 23576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:29.158221 23576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:29.166320 23576 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:36379
I20260929 03:10:29.166337 23643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:36379 every 8 connection(s)
I20260929 03:10:29.167454 23576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:29.170734 23644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:29.171536 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 23576
I20260929 03:10:29.171674 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:29.171947 21177 external_mini_cluster.cc:1428] Setting key 233225db44a6957f208a761cdaba9fea
I20260929 03:10:29.178009 23644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475: Bootstrap starting.
I20260929 03:10:29.180932 23644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:29.181756 23644 log.cc:824] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:10:29 dist-test-slave-k1dg krb5kdc[23563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:29.184325 23644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475: No bootstrap required, opened a new log
I20260929 03:10:29.188266 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:29.173507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.20.174.126:36379)
0929 03:10:29.173978 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:29.173998 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:29.174046 (+ 48us) server_negotiation.cc:407] Connection header received
0929 03:10:29.174919 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:29.174944 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:29.175389 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:29.175837 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:29.176559 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.178103 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.179246 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.179964 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.180135 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:29.182903 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:29.182910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:29.182928 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:29.182959 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:29.184934 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:29.185387 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:29.185402 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:29.185408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:29.185551 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:29.185816 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:29.185819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:29.185821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:29.186361 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:29.186625 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:29.186856 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 03:10:29.187228 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":115,"threads_started":1}
I20260929 03:10:29.189173 23644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f10f10948be44278c6baf6fb8ce1475" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36379 } }
I20260929 03:10:29.189471 23644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:29.189525 23644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f10f10948be44278c6baf6fb8ce1475, State: Initialized, Role: FOLLOWER
I20260929 03:10:29.190014 23644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f10f10948be44278c6baf6fb8ce1475" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36379 } }
I20260929 03:10:29.190181 23644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:29.190243 23644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:29.190339 23644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:29.191437 23644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f10f10948be44278c6baf6fb8ce1475" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36379 } }
I20260929 03:10:29.191751 23644 leader_election.cc:302] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f10f10948be44278c6baf6fb8ce1475; no voters:
I20260929 03:10:29.192026 23644 leader_election.cc:288] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:29.192170 23649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:29.192432 23649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [term 1 LEADER]: Becoming Leader. State: Replica: 0f10f10948be44278c6baf6fb8ce1475, State: Running, Role: LEADER
I20260929 03:10:29.192818 23649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f10f10948be44278c6baf6fb8ce1475" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36379 } }
I20260929 03:10:29.193349 23644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:29.194555 23651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f10f10948be44278c6baf6fb8ce1475. Latest consensus state: current_term: 1 leader_uuid: "0f10f10948be44278c6baf6fb8ce1475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f10f10948be44278c6baf6fb8ce1475" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36379 } } }
I20260929 03:10:29.194687 23651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:29.195206 23650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f10f10948be44278c6baf6fb8ce1475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f10f10948be44278c6baf6fb8ce1475" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36379 } } }
I20260929 03:10:29.195362 23650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475 [sys.catalog]: This master's current role is: LEADER
W20260929 03:10:29.198115 23664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:10:29.198189 23664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:10:29.198282 23658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:29.199096 23658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:29.205410 23658 catalog_manager.cc:1347] Generated new cluster ID: 102b7a77acc84cc198b062a7451d8d25
I20260929 03:10:29.205513 23658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:29.215426 23658 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:29.216584 23658 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:29.230283 23658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f10f10948be44278c6baf6fb8ce1475: Generated new TSK 0
I20260929 03:10:29.231123 23658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:10:29.266021 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36379
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:29.385320 23670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:29.385699 23670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:29.385829 23670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:29.391175 23670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:29.391275 23670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:29.391379 23670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:29.397332 23670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:29.398581 23670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:29.399730 23670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:29.410578 23678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:29.410831 23676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:29.410619 23675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:29.410781 23670 server_base.cc:1034] running on GCE node
I20260929 03:10:29.411382 23670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:29.412111 23670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:29.413362 23670 hybrid_clock.cc:648] HybridClock initialized: now 1790651429413339 us; error 47 us; skew 500 ppm
Sep 29 03:10:29 dist-test-slave-k1dg krb5kdc[23563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:29.416671 23670 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:10:29.417770 23670 webserver.cc:460] Webserver started at http://127.20.174.65:44949/ using document root <none> and password file <none>
I20260929 03:10:29.418373 23670 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:29.418504 23670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:29.418747 23670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:29.420878 23670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "071b9e5f228a4f258b0b46daac16b6a1"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "17f9dc1d2df69948161e2aad4f888b99"
server_key_iv: "febe2293f49950bbee9f9888c0ad5a1d"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.421469 23670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "071b9e5f228a4f258b0b46daac16b6a1"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "17f9dc1d2df69948161e2aad4f888b99"
server_key_iv: "febe2293f49950bbee9f9888c0ad5a1d"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.425616 23670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 03:10:29.428680 23685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:29.430107 23670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:29.430274 23670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "071b9e5f228a4f258b0b46daac16b6a1"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "17f9dc1d2df69948161e2aad4f888b99"
server_key_iv: "febe2293f49950bbee9f9888c0ad5a1d"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.430438 23670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:29.450582 23670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:29.454115 23670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:29.454419 23670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:29.455124 23670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:29.456051 23670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:29.456135 23670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:29.456209 23670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:29.456252 23670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:29.465891 23670 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:39827
I20260929 03:10:29.465914 23798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:39827 every 8 connection(s)
I20260929 03:10:29.466889 23670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:29.473601 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 23670
I20260929 03:10:29.473744 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:29.474027 21177 external_mini_cluster.cc:1428] Setting key 3dd3f63707dcb3623c34008765a2a1b3
Sep 29 03:10:29 dist-test-slave-k1dg krb5kdc[23563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:29.483579 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:29.468906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:59149 (local address 127.20.174.126:36379)
0929 03:10:29.469112 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:29.469116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:29.470224 (+ 1108us) server_negotiation.cc:407] Connection header received
0929 03:10:29.471299 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:29.471303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:29.471364 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:29.471480 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:29.472772 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.473378 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.474879 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.475200 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.475313 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:29.478670 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:29.478677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:29.478683 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:29.478722 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:29.480843 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:29.481470 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:29.481476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:29.481481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:29.481598 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:29.482084 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:29.482090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:29.482094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:29.482364 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:29.482510 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:29.483211 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 03:10:29.483374 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 03:10:29.484422 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:29.469372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:36379 (local address 127.20.174.65:59149)
0929 03:10:29.470057 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:29.470103 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:29.471049 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:29.471686 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:29.471694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:29.472081 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:29.472596 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:29.472610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.473549 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.473552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:29.474751 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:29.474763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.475472 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.475477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:29.475665 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:29.476397 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:29.476415 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:29.478378 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:29.481006 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:29.481013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:29.481027 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:29.481330 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:29.481784 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:29.481787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:29.481788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:29.481954 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:29.482566 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:29.482573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:29.482914 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 03:10:29.483091 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 03:10:29.483387 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:10:29.485870 23799 heartbeater.cc:347] Connected to a master server at 127.20.174.126:36379
I20260929 03:10:29.486146 23799 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:29.486766 23799 heartbeater.cc:511] Master 127.20.174.126:36379 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:10:29.488502 23608 ts_manager.cc:194] Registered new tserver with Master: 071b9e5f228a4f258b0b46daac16b6a1 (127.20.174.65:39827)
I20260929 03:10:29.490254 23608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:59149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:10:29.506016 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36379
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:29.620951 23804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:29.621233 23804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:29.621277 23804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:29.625512 23804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:29.625584 23804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:29.625646 23804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:29.631125 23804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:29.632299 23804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:29.633445 23804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:29.644290 23810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:29.644301 23812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:29.644408 23809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:29.644649 23804 server_base.cc:1034] running on GCE node
I20260929 03:10:29.645118 23804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:29.645810 23804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:29.647022 23804 hybrid_clock.cc:648] HybridClock initialized: now 1790651429647002 us; error 33 us; skew 500 ppm
Sep 29 03:10:29 dist-test-slave-k1dg krb5kdc[23563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:29.649964 23804 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:10:29.651007 23804 webserver.cc:460] Webserver started at http://127.20.174.66:45515/ using document root <none> and password file <none>
I20260929 03:10:29.651600 23804 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:29.651700 23804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:29.651981 23804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:29.654100 23804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "106d5acbab944b2fb898b2032a6eb901"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "d153f8c523eece425d77b0c122ec81fb"
server_key_iv: "74c2463a3dfe608f5d67afa7e3090ad3"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.654883 23804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "106d5acbab944b2fb898b2032a6eb901"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "d153f8c523eece425d77b0c122ec81fb"
server_key_iv: "74c2463a3dfe608f5d67afa7e3090ad3"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.658859 23804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:10:29.661727 23819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:29.663194 23804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:29.663353 23804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "106d5acbab944b2fb898b2032a6eb901"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "d153f8c523eece425d77b0c122ec81fb"
server_key_iv: "74c2463a3dfe608f5d67afa7e3090ad3"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.663484 23804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:29.676100 23804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:29.681192 23804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:29.681456 23804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:29.682387 23804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:29.683619 23804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:29.683687 23804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:29.683746 23804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:29.683779 23804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:29.696046 23804 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:46477
I20260929 03:10:29.696092 23932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:46477 every 8 connection(s)
I20260929 03:10:29.697016 23804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:29.702993 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 23804
I20260929 03:10:29.703145 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:29.703454 21177 external_mini_cluster.cc:1428] Setting key fb79d2ef09c4e468775d9aeb08c6abd1
Sep 29 03:10:29 dist-test-slave-k1dg krb5kdc[23563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:29.712340 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:29.699270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:47763 (local address 127.20.174.126:36379)
0929 03:10:29.699435 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:29.699439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:29.700633 (+ 1194us) server_negotiation.cc:407] Connection header received
0929 03:10:29.701616 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:29.701619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:29.701679 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:29.701789 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:29.703101 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.703761 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.705168 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.705389 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.705509 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:29.708464 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:29.708470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:29.708477 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:29.708503 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:29.710092 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:29.710716 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:29.710720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:29.710722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:29.710808 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:29.711192 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:29.711195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:29.711197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:29.711391 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:29.711521 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:29.712042 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 03:10:29.712172 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:10:29.713286 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:29.699688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:36379 (local address 127.20.174.66:47763)
0929 03:10:29.700454 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:29.700499 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:29.701397 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:29.701969 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:29.701976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:29.702356 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:29.702954 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:29.702968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.703903 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.703907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:29.705044 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:29.705054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.705553 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.705557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:29.705729 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:29.706390 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:29.706406 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:29.708282 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:29.710226 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:29.710232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:29.710248 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:29.710610 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:29.710911 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:29.710914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:29.710916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:29.711076 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:29.711499 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:29.711506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:29.711786 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 03:10:29.711983 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 03:10:29.712339 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":117,"threads_started":1}
I20260929 03:10:29.714792 23933 heartbeater.cc:347] Connected to a master server at 127.20.174.126:36379
I20260929 03:10:29.715112 23933 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:29.715669 23933 heartbeater.cc:511] Master 127.20.174.126:36379 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:10:29.716851 23608 ts_manager.cc:194] Registered new tserver with Master: 106d5acbab944b2fb898b2032a6eb901 (127.20.174.66:46477)
I20260929 03:10:29.717475 23608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:47763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:10:29.734113 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36379
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:29.856546 23938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:29.856895 23938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:29.856990 23938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:29.861508 23938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:29.861624 23938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:29.861722 23938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:29.868075 23938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:29.869246 23938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:29.870325 23938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:29.882539 23946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:29.882534 23944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:29.882529 23943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:29.883023 23938 server_base.cc:1034] running on GCE node
I20260929 03:10:29.883546 23938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:29.884308 23938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:29.885497 23938 hybrid_clock.cc:648] HybridClock initialized: now 1790651429885474 us; error 38 us; skew 500 ppm
Sep 29 03:10:29 dist-test-slave-k1dg krb5kdc[23563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:29.888695 23938 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:10:29.889844 23938 webserver.cc:460] Webserver started at http://127.20.174.67:41987/ using document root <none> and password file <none>
I20260929 03:10:29.890545 23938 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:29.890668 23938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:29.890933 23938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:29.893150 23938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "9780434aa1a64079a572d44893b20b11"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "2d6ca6c66315861f5f612e9e501fb8e7"
server_key_iv: "b4ba89e839629822e34ebad16a6fa4b6"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.893828 23938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "9780434aa1a64079a572d44893b20b11"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "2d6ca6c66315861f5f612e9e501fb8e7"
server_key_iv: "b4ba89e839629822e34ebad16a6fa4b6"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.898142 23938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 03:10:29.900804 23953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:29.902184 23938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:29.902312 23938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "9780434aa1a64079a572d44893b20b11"
format_stamp: "Formatted at 2026-09-29 03:10:29 on dist-test-slave-k1dg"
server_key: "2d6ca6c66315861f5f612e9e501fb8e7"
server_key_iv: "b4ba89e839629822e34ebad16a6fa4b6"
server_key_version: "encryptionkey@0"
I20260929 03:10:29.902478 23938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:29.913185 23938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:29.916785 23938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:29.917016 23938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:29.917682 23938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:29.918660 23938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:29.918710 23938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:29.918776 23938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:29.918807 23938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:29.929730 23938 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:41893
I20260929 03:10:29.929747 24066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:41893 every 8 connection(s)
I20260929 03:10:29.930806 23938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:29.934675 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 23938
I20260929 03:10:29.934859 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:29.935245 21177 external_mini_cluster.cc:1428] Setting key 07468cec493fac35754b04b47a3592cd
Sep 29 03:10:29 dist-test-slave-k1dg krb5kdc[23563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:29.947129 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:29.932882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:53497 (local address 127.20.174.126:36379)
0929 03:10:29.933014 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:29.933018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:29.935081 (+ 2063us) server_negotiation.cc:407] Connection header received
0929 03:10:29.936505 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:29.936508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:29.936566 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:29.936621 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:29.938300 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.938972 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.940201 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.940437 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.940549 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:29.943303 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:29.943310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:29.943315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:29.943350 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:29.944878 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:29.945468 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:29.945472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:29.945474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:29.945558 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:29.945912 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:29.945916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:29.945918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:29.946119 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:29.946229 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:29.946817 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 03:10:29.946948 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:10:29.948084 24069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:29.933506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:36379 (local address 127.20.174.67:53497)
0929 03:10:29.934870 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:29.934937 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:29.936241 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:29.937081 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:29.937089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:29.937469 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:29.938142 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:29.938159 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.939111 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.939114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:29.940055 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:29.940065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:29.940606 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:29.940610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:29.940796 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:29.941521 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:29.941539 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:29.943118 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:29.945025 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:29.945031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:29.945055 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:29.945346 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:29.945655 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:29.945659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:29.945661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:29.945812 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:29.946237 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:29.946247 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:29.946571 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 03:10:29.946773 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 03:10:29.947112 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"thread_start_us":890,"threads_started":1}
I20260929 03:10:29.949429 24067 heartbeater.cc:347] Connected to a master server at 127.20.174.126:36379
I20260929 03:10:29.949725 24067 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:29.950465 24067 heartbeater.cc:511] Master 127.20.174.126:36379 requested a full tablet report, sending...
I20260929 03:10:29.951759 23608 ts_manager.cc:194] Registered new tserver with Master: 9780434aa1a64079a572d44893b20b11 (127.20.174.67:41893)
I20260929 03:10:29.952391 23608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:53497
I20260929 03:10:29.962139 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:10:29.978679 23647 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:29.976881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.20.174.126:36379)
0929 03:10:29.977048 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:29.977053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:29.977210 (+ 157us) server_negotiation.cc:407] Connection header received
0929 03:10:29.977445 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:29.977448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:29.977495 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:29.977584 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:29.978504 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37870: BlockingRecv error: recv got EOF from 127.0.0.1:37870 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 03:10:29.980031 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 23670
I20260929 03:10:29.986040 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 23804
I20260929 03:10:29.992142 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 23938
I20260929 03:10:29.997932 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 23576
2026-09-29T03:10:30Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2231 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:30 dist-test-slave-k1dg krb5kdc[24079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:30 dist-test-slave-k1dg krb5kdc[24079](info): set up 2 sockets
Sep 29 03:10:30 dist-test-slave-k1dg krb5kdc[24079](info): commencing operation
krb5kdc: starting...
W20260929 03:10:31.050700 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:31 dist-test-slave-k1dg krb5kdc[24079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:31.182742 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:43819
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:43819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:31.298633 24092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:31.298975 24092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:31.299058 24092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:31.303313 24092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:31.303414 24092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:31.303449 24092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:31.303488 24092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:31.303546 24092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:31.309269 24092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:43819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:43819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:31.310603 24092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:31.311697 24092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:31.321535 24098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:31.321606 24092 server_base.cc:1034] running on GCE node
W20260929 03:10:31.321537 24097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:31.321537 24100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:31.322422 24092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:31.323798 24092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:31.325037 24092 hybrid_clock.cc:648] HybridClock initialized: now 1790651431325001 us; error 54 us; skew 500 ppm
Sep 29 03:10:31 dist-test-slave-k1dg krb5kdc[24079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651431, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:31.336738 24092 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:31.337895 24092 webserver.cc:460] Webserver started at http://127.20.174.126:33683/ using document root <none> and password file <none>
I20260929 03:10:31.338676 24092 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:31.338784 24092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:31.339032 24092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:31.341482 24092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "67896b15160f4101a4a1d03a7841b44c"
format_stamp: "Formatted at 2026-09-29 03:10:31 on dist-test-slave-k1dg"
server_key: "9cd099a699948d81768b8a07ef83fd44"
server_key_iv: "c73382aecfb7b202103b9c1b3b293b55"
server_key_version: "encryptionkey@0"
I20260929 03:10:31.342154 24092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "67896b15160f4101a4a1d03a7841b44c"
format_stamp: "Formatted at 2026-09-29 03:10:31 on dist-test-slave-k1dg"
server_key: "9cd099a699948d81768b8a07ef83fd44"
server_key_iv: "c73382aecfb7b202103b9c1b3b293b55"
server_key_version: "encryptionkey@0"
I20260929 03:10:31.346808 24092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:10:31.349759 24107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:31.351368 24092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:31.351586 24092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "67896b15160f4101a4a1d03a7841b44c"
format_stamp: "Formatted at 2026-09-29 03:10:31 on dist-test-slave-k1dg"
server_key: "9cd099a699948d81768b8a07ef83fd44"
server_key_iv: "c73382aecfb7b202103b9c1b3b293b55"
server_key_version: "encryptionkey@0"
I20260929 03:10:31.351728 24092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:31.373275 24092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:31.377085 24092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:31.377350 24092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:31.385548 24092 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:43819
I20260929 03:10:31.385955 24159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:43819 every 8 connection(s)
I20260929 03:10:31.386778 24092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:31.390060 24160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:31.390333 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 24092
I20260929 03:10:31.390509 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:31.390791 21177 external_mini_cluster.cc:1428] Setting key b6fab38cb3bea7ab5ca1a02dc5a9d76e
I20260929 03:10:31.398347 24160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c: Bootstrap starting.
I20260929 03:10:31.400936 24160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c: Neither blocks nor log segments found. Creating new log.
Sep 29 03:10:31 dist-test-slave-k1dg krb5kdc[24079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:31.401871 24160 log.cc:824] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:31.404322 24160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c: No bootstrap required, opened a new log
I20260929 03:10:31.407128 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:31.392220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.20.174.126:43819)
0929 03:10:31.392761 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:31.392781 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:31.392828 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:10:31.394220 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:31.394250 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:31.394809 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:31.395263 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:31.396248 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:31.397365 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:31.398308 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:31.398967 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:31.399146 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:31.401886 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:31.401899 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:31.401914 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:31.401946 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:31.403993 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:31.404485 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:31.404492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:31.404498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:31.404622 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:31.404943 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:31.404946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:31.404948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:31.405371 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:31.405589 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:31.405927 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 03:10:31.406268 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":166,"threads_started":1}
I20260929 03:10:31.408946 24160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67896b15160f4101a4a1d03a7841b44c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43819 } }
I20260929 03:10:31.409238 24160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:31.409298 24160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67896b15160f4101a4a1d03a7841b44c, State: Initialized, Role: FOLLOWER
I20260929 03:10:31.409850 24160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67896b15160f4101a4a1d03a7841b44c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43819 } }
I20260929 03:10:31.410048 24160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:31.410158 24160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:31.410272 24160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:31.410578 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:10:31.411448 24160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67896b15160f4101a4a1d03a7841b44c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43819 } }
I20260929 03:10:31.411825 24160 leader_election.cc:302] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67896b15160f4101a4a1d03a7841b44c; no voters:
I20260929 03:10:31.412221 24160 leader_election.cc:288] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:31.412647 24167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:31.412906 24167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [term 1 LEADER]: Becoming Leader. State: Replica: 67896b15160f4101a4a1d03a7841b44c, State: Running, Role: LEADER
I20260929 03:10:31.413295 24167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67896b15160f4101a4a1d03a7841b44c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43819 } }
I20260929 03:10:31.413801 24160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:31.414466 24167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67896b15160f4101a4a1d03a7841b44c. Latest consensus state: current_term: 1 leader_uuid: "67896b15160f4101a4a1d03a7841b44c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67896b15160f4101a4a1d03a7841b44c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43819 } } }
I20260929 03:10:31.414698 24167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:31.414896 24168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67896b15160f4101a4a1d03a7841b44c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67896b15160f4101a4a1d03a7841b44c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43819 } } }
I20260929 03:10:31.414986 24168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:31.415160 24170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:31.419777 24170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:31.426710 24170 catalog_manager.cc:1347] Generated new cluster ID: 31adf1776030417b9915a02723b21701
I20260929 03:10:31.426896 24170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:31.435925 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:31.429220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.20.174.126:43819)
0929 03:10:31.429348 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:31.429354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:31.429556 (+ 202us) server_negotiation.cc:407] Connection header received
0929 03:10:31.429741 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:31.429744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:31.429792 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:31.429842 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:31.430629 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:31.431536 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:31.432853 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:31.433266 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:31.433337 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:31.434642 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:31.434646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:31.434649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:31.435325 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:31.435365 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:31.435400 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:31.435473 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:31.435681 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 03:10:31.435755 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:10:31.448136 24170 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:31.449532 24170 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:31.454979 24170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67896b15160f4101a4a1d03a7841b44c: Generated new TSK 0
I20260929 03:10:31.455844 24170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:10:31.467568 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 24092
2026-09-29T03:10:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1466 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:31 dist-test-slave-k1dg krb5kdc[24193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:31 dist-test-slave-k1dg krb5kdc[24193](info): set up 2 sockets
Sep 29 03:10:31 dist-test-slave-k1dg krb5kdc[24193](info): commencing operation
krb5kdc: starting...
W20260929 03:10:32.510828 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:32 dist-test-slave-k1dg krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:32.640254 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40885
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:32.760001 24206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:32.760352 24206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:32.760430 24206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:32.764642 24206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:32.764765 24206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:32.764802 24206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:32.764844 24206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:32.764878 24206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:32.770609 24206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:40885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:32.773320 24206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:32.774636 24206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:32.783716 24211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:32.783754 24206 server_base.cc:1034] running on GCE node
W20260929 03:10:32.783716 24214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:32.785480 24212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:32.786391 24206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:32.788158 24206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:32.789328 24206 hybrid_clock.cc:648] HybridClock initialized: now 1790651432789305 us; error 58 us; skew 500 ppm
Sep 29 03:10:32 dist-test-slave-k1dg krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651432, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:32.793066 24206 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:32.794014 24206 webserver.cc:460] Webserver started at http://127.20.174.126:37181/ using document root <none> and password file <none>
I20260929 03:10:32.794751 24206 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:32.794852 24206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:32.795130 24206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:32.797508 24206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "5981f2a63bcc4052ac83b5682f240842"
format_stamp: "Formatted at 2026-09-29 03:10:32 on dist-test-slave-k1dg"
server_key: "26a59f80d4852d92242217f476e56379"
server_key_iv: "360d2b89d37487d5274bf3e84cc28b62"
server_key_version: "encryptionkey@0"
I20260929 03:10:32.798099 24206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "5981f2a63bcc4052ac83b5682f240842"
format_stamp: "Formatted at 2026-09-29 03:10:32 on dist-test-slave-k1dg"
server_key: "26a59f80d4852d92242217f476e56379"
server_key_iv: "360d2b89d37487d5274bf3e84cc28b62"
server_key_version: "encryptionkey@0"
I20260929 03:10:32.802035 24206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:10:32.805065 24221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:32.806339 24206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:10:32.806553 24206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "5981f2a63bcc4052ac83b5682f240842"
format_stamp: "Formatted at 2026-09-29 03:10:32 on dist-test-slave-k1dg"
server_key: "26a59f80d4852d92242217f476e56379"
server_key_iv: "360d2b89d37487d5274bf3e84cc28b62"
server_key_version: "encryptionkey@0"
I20260929 03:10:32.806674 24206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:32.830024 24206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:32.833176 24206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:32.833395 24206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:32.841289 24206 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:40885
I20260929 03:10:32.841610 24273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:40885 every 8 connection(s)
I20260929 03:10:32.842308 24206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:32.845505 24274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:32.847395 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 24206
I20260929 03:10:32.847503 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:32.847797 21177 external_mini_cluster.cc:1428] Setting key 0c8fb5aafeaf07b80e083dde5ccf4953
I20260929 03:10:32.852324 24274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842: Bootstrap starting.
I20260929 03:10:32.854985 24274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:32.855742 24274 log.cc:824] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:10:32 dist-test-slave-k1dg krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:32.857788 24274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842: No bootstrap required, opened a new log
I20260929 03:10:32.861713 24274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5981f2a63bcc4052ac83b5682f240842" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40885 } }
I20260929 03:10:32.862054 24274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:32.862133 24274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5981f2a63bcc4052ac83b5682f240842, State: Initialized, Role: FOLLOWER
I20260929 03:10:32.862691 24274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5981f2a63bcc4052ac83b5682f240842" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40885 } }
I20260929 03:10:32.862833 24274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:32.862921 24274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:32.863044 24274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:32.863292 24277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:32.849338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.20.174.126:40885)
0929 03:10:32.849827 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:32.849845 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:32.849882 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:10:32.850636 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:32.850655 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:32.850994 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:32.851384 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:32.852172 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:32.853199 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:32.854237 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:32.854863 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:32.855051 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:32.857662 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:32.857678 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:32.857692 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:32.857720 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:32.859725 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:32.860279 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:32.860285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:32.860290 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:32.860415 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:32.860739 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:32.860743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:32.860745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:32.861191 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:32.861429 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:32.861756 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 03:10:32.862258 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":141,"threads_started":1}
I20260929 03:10:32.864146 24274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5981f2a63bcc4052ac83b5682f240842" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40885 } }
I20260929 03:10:32.864521 24274 leader_election.cc:302] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5981f2a63bcc4052ac83b5682f240842; no voters:
I20260929 03:10:32.864900 24274 leader_election.cc:288] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:32.864989 24279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:32.865171 24279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [term 1 LEADER]: Becoming Leader. State: Replica: 5981f2a63bcc4052ac83b5682f240842, State: Running, Role: LEADER
I20260929 03:10:32.865573 24279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5981f2a63bcc4052ac83b5682f240842" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40885 } }
I20260929 03:10:32.866312 24274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:32.866945 24279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5981f2a63bcc4052ac83b5682f240842. Latest consensus state: current_term: 1 leader_uuid: "5981f2a63bcc4052ac83b5682f240842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5981f2a63bcc4052ac83b5682f240842" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40885 } } }
I20260929 03:10:32.867071 24279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:32.867393 24280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5981f2a63bcc4052ac83b5682f240842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5981f2a63bcc4052ac83b5682f240842" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40885 } } }
I20260929 03:10:32.867588 24280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:32.867637 24284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:32.871054 24284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:32.877055 24284 catalog_manager.cc:1347] Generated new cluster ID: 7311c41d0c6042a0a0b82730cedec28c
I20260929 03:10:32.877147 24284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:32.887372 24284 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:32.888746 24284 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:32.896661 24284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5981f2a63bcc4052ac83b5682f240842: Generated new TSK 0
I20260929 03:10:32.897545 24284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:10:32.929522 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40885
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:33.049948 24300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:33.050235 24300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:33.050282 24300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:33.054560 24300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:33.054633 24300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:33.054695 24300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:33.060286 24300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:33.061455 24300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:33.062614 24300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:33.073412 24305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:33.073434 24306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:33.073493 24300 server_base.cc:1034] running on GCE node
W20260929 03:10:33.073433 24308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:33.074196 24300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:33.075017 24300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:33.076212 24300 hybrid_clock.cc:648] HybridClock initialized: now 1790651433076187 us; error 44 us; skew 500 ppm
Sep 29 03:10:33 dist-test-slave-k1dg krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:33.079453 24300 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:10:33.080462 24300 webserver.cc:460] Webserver started at http://127.20.174.65:33011/ using document root <none> and password file <none>
I20260929 03:10:33.081094 24300 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:33.081231 24300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:33.081521 24300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:33.083791 24300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "d5f83a8256af4127a54d8b386dc1a41f"
format_stamp: "Formatted at 2026-09-29 03:10:33 on dist-test-slave-k1dg"
server_key: "bf7118c46a5f28b943eafd8615c1cbd1"
server_key_iv: "2d732322cdfcafa9dd244f10c1b2d7ce"
server_key_version: "encryptionkey@0"
I20260929 03:10:33.084422 24300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "d5f83a8256af4127a54d8b386dc1a41f"
format_stamp: "Formatted at 2026-09-29 03:10:33 on dist-test-slave-k1dg"
server_key: "bf7118c46a5f28b943eafd8615c1cbd1"
server_key_iv: "2d732322cdfcafa9dd244f10c1b2d7ce"
server_key_version: "encryptionkey@0"
I20260929 03:10:33.088785 24300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:33.091466 24315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:33.092937 24300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:33.093123 24300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "d5f83a8256af4127a54d8b386dc1a41f"
format_stamp: "Formatted at 2026-09-29 03:10:33 on dist-test-slave-k1dg"
server_key: "bf7118c46a5f28b943eafd8615c1cbd1"
server_key_iv: "2d732322cdfcafa9dd244f10c1b2d7ce"
server_key_version: "encryptionkey@0"
I20260929 03:10:33.093314 24300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:33.120968 24300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:33.124388 24300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:33.124614 24300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:33.125267 24300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:33.126201 24300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:33.126282 24300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:33.126348 24300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:33.126401 24300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:33.135958 24300 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:35425
I20260929 03:10:33.135977 24428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:35425 every 8 connection(s)
I20260929 03:10:33.136901 24300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:33.146979 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 24300
I20260929 03:10:33.147082 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:33.147373 21177 external_mini_cluster.cc:1428] Setting key 955b32ee4075029369c0d7ac3febe1fb
Sep 29 03:10:33 dist-test-slave-k1dg krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:33.152601 24277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:33.139032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:33083 (local address 127.20.174.126:40885)
0929 03:10:33.139185 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:33.139189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:33.140156 (+ 967us) server_negotiation.cc:407] Connection header received
0929 03:10:33.141581 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:33.141585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:33.141647 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:33.141755 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:33.143541 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.144091 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.145337 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.145581 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.145700 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:33.148528 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:33.148532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:33.148534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:33.148569 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:33.150148 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:33.150810 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:33.150814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:33.150816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:33.150905 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:33.151293 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:33.151297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:33.151299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:33.151502 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:33.151618 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:33.152342 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 03:10:33.152463 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:10:33.153885 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:33.139413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40885 (local address 127.20.174.65:33083)
0929 03:10:33.139947 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:33.140034 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:33.141346 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:33.141941 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:33.141948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:33.142520 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:33.143353 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:33.143395 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.144229 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.144232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:33.145192 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:33.145203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.145745 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.145749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:33.145946 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:33.146652 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:33.146669 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:33.148341 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:33.150291 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:33.150297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:33.150314 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:33.150695 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:33.151014 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:33.151017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:33.151019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:33.151184 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:33.151617 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:33.151625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:33.151981 (+ 356us) client_negotiation.cc:769] Sending connection context
0929 03:10:33.152207 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 03:10:33.152613 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
I20260929 03:10:33.155486 24429 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40885
I20260929 03:10:33.155759 24429 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:33.156317 24429 heartbeater.cc:511] Master 127.20.174.126:40885 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:10:33.158031 24238 ts_manager.cc:194] Registered new tserver with Master: d5f83a8256af4127a54d8b386dc1a41f (127.20.174.65:35425)
I20260929 03:10:33.159266 24238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:33083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:10:33.178411 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40885
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:33.295243 24434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:33.295560 24434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:33.295643 24434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:33.300100 24434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:33.300197 24434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:33.300293 24434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:33.305948 24434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:33.307132 24434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:33.308235 24434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:33.318499 24439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:33.318507 24442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:33.318548 24440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:33.318524 24434 server_base.cc:1034] running on GCE node
I20260929 03:10:33.319280 24434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:33.319969 24434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:33.321146 24434 hybrid_clock.cc:648] HybridClock initialized: now 1790651433321131 us; error 39 us; skew 500 ppm
Sep 29 03:10:33 dist-test-slave-k1dg krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:33.324234 24434 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:10:33.325316 24434 webserver.cc:460] Webserver started at http://127.20.174.66:44057/ using document root <none> and password file <none>
I20260929 03:10:33.325910 24434 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:33.326004 24434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:33.326241 24434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:33.328420 24434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "63632b4556c648aeba8070f4d04f61aa"
format_stamp: "Formatted at 2026-09-29 03:10:33 on dist-test-slave-k1dg"
server_key: "92b75382ea9f939234473efd8aaadaba"
server_key_iv: "6e4743173e55319f842758c2e7656e29"
server_key_version: "encryptionkey@0"
I20260929 03:10:33.329010 24434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "63632b4556c648aeba8070f4d04f61aa"
format_stamp: "Formatted at 2026-09-29 03:10:33 on dist-test-slave-k1dg"
server_key: "92b75382ea9f939234473efd8aaadaba"
server_key_iv: "6e4743173e55319f842758c2e7656e29"
server_key_version: "encryptionkey@0"
I20260929 03:10:33.332885 24434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 03:10:33.336150 24449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:33.337606 24434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:33.337728 24434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "63632b4556c648aeba8070f4d04f61aa"
format_stamp: "Formatted at 2026-09-29 03:10:33 on dist-test-slave-k1dg"
server_key: "92b75382ea9f939234473efd8aaadaba"
server_key_iv: "6e4743173e55319f842758c2e7656e29"
server_key_version: "encryptionkey@0"
I20260929 03:10:33.337894 24434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:33.351336 24434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:33.354720 24434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:33.354936 24434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:33.355645 24434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:33.356515 24434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:33.356565 24434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:33.356603 24434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:33.356619 24434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:33.366173 24434 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:41335
I20260929 03:10:33.366204 24562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:41335 every 8 connection(s)
I20260929 03:10:33.367347 24434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:33.375695 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 24434
I20260929 03:10:33.375788 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:33.376078 21177 external_mini_cluster.cc:1428] Setting key b89d79a8c0b5b9b81e6d14d7a080f090
Sep 29 03:10:33 dist-test-slave-k1dg krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:33.382160 24277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:33.369458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:48241 (local address 127.20.174.126:40885)
0929 03:10:33.369597 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:33.369600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:33.370487 (+ 887us) server_negotiation.cc:407] Connection header received
0929 03:10:33.371937 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:33.371940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:33.371997 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:33.372077 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:33.373311 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.373871 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.375114 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.375343 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.375472 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:33.378162 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:33.378166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:33.378169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:33.378201 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:33.380007 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:33.380571 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:33.380577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:33.380584 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:33.380663 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:33.381017 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:33.381020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:33.381022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:33.381213 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:33.381323 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:33.381846 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 03:10:33.381994 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:10:33.383073 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:33.369864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40885 (local address 127.20.174.66:48241)
0929 03:10:33.370313 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:33.370355 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:33.371713 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:33.372258 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:33.372265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:33.372659 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:33.373151 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:33.373166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.374012 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.374015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:33.374986 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:33.374995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.375514 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.375518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:33.375691 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:33.376401 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:33.376420 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:33.377987 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:33.380142 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:33.380148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:33.380161 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:33.380447 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:33.380768 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:33.380775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:33.380777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:33.380913 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:33.381323 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:33.381331 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:33.381607 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 03:10:33.381806 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 03:10:33.382143 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260929 03:10:33.384428 24563 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40885
I20260929 03:10:33.384677 24563 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:33.385222 24563 heartbeater.cc:511] Master 127.20.174.126:40885 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:10:33.386379 24238 ts_manager.cc:194] Registered new tserver with Master: 63632b4556c648aeba8070f4d04f61aa (127.20.174.66:41335)
I20260929 03:10:33.387068 24238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:48241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:10:33.405892 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40885
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:33.524668 24568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:33.524972 24568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:33.525018 24568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:33.529363 24568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:33.529453 24568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:33.529517 24568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:33.535400 24568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:33.536623 24568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:33.537746 24568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:33.548045 24576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:33.548045 24574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:33.548137 24573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:33.548568 24568 server_base.cc:1034] running on GCE node
I20260929 03:10:33.549075 24568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:33.549832 24568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:33.551041 24568 hybrid_clock.cc:648] HybridClock initialized: now 1790651433551021 us; error 32 us; skew 500 ppm
Sep 29 03:10:33 dist-test-slave-k1dg krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:33.553962 24568 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:10:33.555022 24568 webserver.cc:460] Webserver started at http://127.20.174.67:33153/ using document root <none> and password file <none>
I20260929 03:10:33.555562 24568 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:33.555621 24568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:33.555789 24568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:33.557859 24568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "bd8f6c25361247139566b5163ca3defd"
format_stamp: "Formatted at 2026-09-29 03:10:33 on dist-test-slave-k1dg"
server_key: "8dd8a49308e2748bac8ade588509f22d"
server_key_iv: "38a247806c5959709dabb65f5e786f6a"
server_key_version: "encryptionkey@0"
I20260929 03:10:33.558395 24568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "bd8f6c25361247139566b5163ca3defd"
format_stamp: "Formatted at 2026-09-29 03:10:33 on dist-test-slave-k1dg"
server_key: "8dd8a49308e2748bac8ade588509f22d"
server_key_iv: "38a247806c5959709dabb65f5e786f6a"
server_key_version: "encryptionkey@0"
I20260929 03:10:33.562371 24568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:10:33.564982 24583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:33.566349 24568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:10:33.566527 24568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "bd8f6c25361247139566b5163ca3defd"
format_stamp: "Formatted at 2026-09-29 03:10:33 on dist-test-slave-k1dg"
server_key: "8dd8a49308e2748bac8ade588509f22d"
server_key_iv: "38a247806c5959709dabb65f5e786f6a"
server_key_version: "encryptionkey@0"
I20260929 03:10:33.566654 24568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:33.588491 24568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:33.592262 24568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:33.592682 24568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:33.593506 24568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:33.594528 24568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:33.594601 24568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:33.594674 24568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:33.594722 24568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:33.604048 24568 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:40983
I20260929 03:10:33.604067 24696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:40983 every 8 connection(s)
I20260929 03:10:33.604987 24568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:33.613176 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 24568
I20260929 03:10:33.613328 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:33.613679 21177 external_mini_cluster.cc:1428] Setting key a7f28eb922c85ea186a0f472af23d807
Sep 29 03:10:33 dist-test-slave-k1dg krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651433, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:33.619577 24277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:33.606847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:55965 (local address 127.20.174.126:40885)
0929 03:10:33.606989 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:33.606993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:33.607914 (+ 921us) server_negotiation.cc:407] Connection header received
0929 03:10:33.609345 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:33.609348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:33.609396 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:33.609507 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:33.610868 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.611508 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.612726 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.612961 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.613042 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:33.616219 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:33.616223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:33.616226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:33.616259 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:33.617569 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:33.618108 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:33.618111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:33.618112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:33.618187 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:33.618561 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:33.618564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:33.618565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:33.618741 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:33.618820 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:33.619358 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 03:10:33.619460 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:10:33.620544 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:33.607242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40885 (local address 127.20.174.67:55965)
0929 03:10:33.607675 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:33.607735 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:33.609126 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:33.609670 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:33.609677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:33.610122 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:33.610718 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:33.610733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.611647 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.611650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:33.612606 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:33.612617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.613147 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.613154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:33.613401 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:33.614114 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:33.614131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:33.616031 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:33.617677 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:33.617682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:33.617696 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:33.618013 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:33.618276 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:33.618278 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:33.618279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:33.618424 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:33.618868 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:33.618874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:33.619132 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 03:10:33.619305 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 03:10:33.619585 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260929 03:10:33.621860 24697 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40885
I20260929 03:10:33.622143 24697 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:33.622702 24697 heartbeater.cc:511] Master 127.20.174.126:40885 requested a full tablet report, sending...
I20260929 03:10:33.623750 24238 ts_manager.cc:194] Registered new tserver with Master: bd8f6c25361247139566b5163ca3defd (127.20.174.67:40983)
I20260929 03:10:33.624353 24238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:55965
I20260929 03:10:33.628103 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:10:33 dist-test-slave-k1dg krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651433, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 03:10:33 dist-test-slave-k1dg krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651433, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:33.654590 24277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:33.645736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.20.174.126:40885)
0929 03:10:33.645913 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:33.645916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:33.646058 (+ 142us) server_negotiation.cc:407] Connection header received
0929 03:10:33.646182 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:33.646184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:33.646231 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:33.646305 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:33.647145 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.647676 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.648641 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:33.648858 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:33.648937 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:33.651545 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:33.651549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:33.651551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:33.651575 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:33.652809 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:33.653250 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:33.653253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:33.653255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:33.653331 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:33.653618 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:33.653621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:33.653623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:33.653904 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:33.654036 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:33.654273 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 03:10:33.654374 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 03:10:33.655098 24238 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36054
I20260929 03:10:33.656013 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 24300
I20260929 03:10:33.661408 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 24434
I20260929 03:10:33.667047 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 24568
I20260929 03:10:33.672575 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 24206
2026-09-29T03:10:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2207 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:33 dist-test-slave-k1dg krb5kdc[24709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:33 dist-test-slave-k1dg krb5kdc[24709](info): set up 2 sockets
Sep 29 03:10:33 dist-test-slave-k1dg krb5kdc[24709](info): commencing operation
krb5kdc: starting...
W20260929 03:10:34.823000 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:34 dist-test-slave-k1dg krb5kdc[24709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:34.951807 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:32991
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:32991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:35.068397 24722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:35.068688 24722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:35.068732 24722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:35.073076 24722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:35.073159 24722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:35.073180 24722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:35.073197 24722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:35.073213 24722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:35.079135 24722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:32991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:32991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:35.080317 24722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:35.081437 24722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:35.090622 24730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:35.090655 24727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:35.090624 24728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:35.091070 24722 server_base.cc:1034] running on GCE node
I20260929 03:10:35.091666 24722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:35.092875 24722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:35.094062 24722 hybrid_clock.cc:648] HybridClock initialized: now 1790651435094044 us; error 37 us; skew 500 ppm
Sep 29 03:10:35 dist-test-slave-k1dg krb5kdc[24709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651435, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:35.097100 24722 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:35.098057 24722 webserver.cc:460] Webserver started at http://127.20.174.126:39271/ using document root <none> and password file <none>
I20260929 03:10:35.098673 24722 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:35.098737 24722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:35.098909 24722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:35.101033 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "910b25446fd441df92d7e7e835da01b8"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "630174aa710143026222b29969b40bc6"
server_key_iv: "1f2aa37abd7785b048e323eb89edde07"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.101573 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "910b25446fd441df92d7e7e835da01b8"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "630174aa710143026222b29969b40bc6"
server_key_iv: "1f2aa37abd7785b048e323eb89edde07"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.105696 24722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:10:35.108484 24737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:35.109817 24722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:35.110015 24722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "910b25446fd441df92d7e7e835da01b8"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "630174aa710143026222b29969b40bc6"
server_key_iv: "1f2aa37abd7785b048e323eb89edde07"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.110127 24722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:35.135341 24722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:35.138803 24722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:35.139026 24722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:35.146667 24722 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:32991
I20260929 03:10:35.146699 24789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:32991 every 8 connection(s)
I20260929 03:10:35.147672 24722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:35.148981 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 24722
I20260929 03:10:35.149111 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:35.149406 21177 external_mini_cluster.cc:1428] Setting key 492b5e805b2b6928480898b3439e21ec
I20260929 03:10:35.151180 24790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:35.159907 24790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8: Bootstrap starting.
Sep 29 03:10:35 dist-test-slave-k1dg krb5kdc[24709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:35.163738 24790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:35.165140 24790 log.cc:824] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:35.168792 24790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8: No bootstrap required, opened a new log
I20260929 03:10:35.169472 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:35.150992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47282 (local address 127.20.174.126:32991)
0929 03:10:35.151897 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:35.151918 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:35.151965 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:10:35.152861 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:35.152886 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:35.153342 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:35.153920 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:35.155117 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.156855 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.158162 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.158959 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.159124 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:35.162565 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:35.162575 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:35.162598 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:35.162644 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:35.165337 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.166118 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.166128 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.166135 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.166292 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.166750 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.166755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.166756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.167372 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:35.167597 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:35.168027 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 03:10:35.168367 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":157,"threads_started":1}
I20260929 03:10:35.173344 24790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910b25446fd441df92d7e7e835da01b8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 32991 } }
I20260929 03:10:35.173664 24790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:35.173780 24790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910b25446fd441df92d7e7e835da01b8, State: Initialized, Role: FOLLOWER
I20260929 03:10:35.174331 24790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910b25446fd441df92d7e7e835da01b8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 32991 } }
I20260929 03:10:35.174561 24790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:35.174631 24790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:35.174746 24790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:35.176272 24790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910b25446fd441df92d7e7e835da01b8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 32991 } }
I20260929 03:10:35.176668 24790 leader_election.cc:302] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 910b25446fd441df92d7e7e835da01b8; no voters:
I20260929 03:10:35.177062 24790 leader_election.cc:288] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:35.177208 24795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:35.177526 24795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [term 1 LEADER]: Becoming Leader. State: Replica: 910b25446fd441df92d7e7e835da01b8, State: Running, Role: LEADER
I20260929 03:10:35.177893 24795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910b25446fd441df92d7e7e835da01b8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 32991 } }
I20260929 03:10:35.178557 24790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:35.179769 24796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "910b25446fd441df92d7e7e835da01b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910b25446fd441df92d7e7e835da01b8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 32991 } } }
I20260929 03:10:35.179908 24796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:35.180280 24797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 910b25446fd441df92d7e7e835da01b8. Latest consensus state: current_term: 1 leader_uuid: "910b25446fd441df92d7e7e835da01b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910b25446fd441df92d7e7e835da01b8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 32991 } } }
I20260929 03:10:35.180361 24804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:35.180474 24797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:35.184538 24804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:35.190696 24804 catalog_manager.cc:1347] Generated new cluster ID: 8edc0f4c26d24f2b9b058e626db88062
I20260929 03:10:35.190784 24804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:35.198891 24804 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:35.201289 24804 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:35.208631 24804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 910b25446fd441df92d7e7e835da01b8: Generated new TSK 0
I20260929 03:10:35.209772 24804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:10:35.245785 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:32991
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:35.368736 24816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:35.369042 24816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:35.369087 24816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:35.373368 24816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:35.373447 24816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:35.373510 24816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:35.379314 24816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:35.380532 24816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:35.381803 24816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:35.392199 24821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:35.392194 24822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:35.392263 24824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:35.392524 24816 server_base.cc:1034] running on GCE node
I20260929 03:10:35.393124 24816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:35.393812 24816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:35.395046 24816 hybrid_clock.cc:648] HybridClock initialized: now 1790651435395013 us; error 59 us; skew 500 ppm
Sep 29 03:10:35 dist-test-slave-k1dg krb5kdc[24709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651435, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:35.398247 24816 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:10:35.399459 24816 webserver.cc:460] Webserver started at http://127.20.174.65:39231/ using document root <none> and password file <none>
I20260929 03:10:35.400138 24816 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:35.400247 24816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:35.400519 24816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:35.403100 24816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "d31ca7e4979e4bf39a9fa86fb7c57bf3"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "daaddcdf202e907a439b1dd7cf1c1882"
server_key_iv: "0e2d9dc51a2b2adcea4887145175d9fc"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.403785 24816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "d31ca7e4979e4bf39a9fa86fb7c57bf3"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "daaddcdf202e907a439b1dd7cf1c1882"
server_key_iv: "0e2d9dc51a2b2adcea4887145175d9fc"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.408252 24816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:35.412009 24831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:35.414810 24816 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:35.414979 24816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "d31ca7e4979e4bf39a9fa86fb7c57bf3"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "daaddcdf202e907a439b1dd7cf1c1882"
server_key_iv: "0e2d9dc51a2b2adcea4887145175d9fc"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.415148 24816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:35.426872 24816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:35.430745 24816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:35.430974 24816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:35.431639 24816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:35.432564 24816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:35.432641 24816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:35.432713 24816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:35.432765 24816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:35.442407 24816 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:45165
I20260929 03:10:35.442520 24944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:45165 every 8 connection(s)
I20260929 03:10:35.443466 24816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:35.443674 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 24816
I20260929 03:10:35.443801 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:35.444132 21177 external_mini_cluster.cc:1428] Setting key f087f6f50a04ba5069b137fde53632a8
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
Sep 29 03:10:35 dist-test-slave-k1dg krb5kdc[24709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651435, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:10:35.459792 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:35.446042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:48121 (local address 127.20.174.126:32991)
0929 03:10:35.446205 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:35.446210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:35.447389 (+ 1179us) server_negotiation.cc:407] Connection header received
0929 03:10:35.449010 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:35.449013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:35.449076 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:35.449203 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:35.450647 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.451318 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.452522 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.452779 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.452903 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:35.455562 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:35.455570 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:35.455573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:35.455600 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:35.457280 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.457864 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.457868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.457870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.457966 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.458381 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.458383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.458385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.458649 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:35.458733 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:35.459422 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 03:10:35.459550 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:10:35.460701 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:35.446673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:32991 (local address 127.20.174.65:48121)
0929 03:10:35.447205 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:35.447264 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:35.448686 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:35.449447 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:35.449454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:35.449886 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:35.450412 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:35.450484 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.451449 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.451453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:35.452381 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:35.452392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.452950 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.452954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:35.453131 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:35.453791 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:35.453808 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:35.455368 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:35.457448 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:35.457455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:35.457471 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:35.457746 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:35.458073 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:35.458076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:35.458078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:35.458251 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:35.458764 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:35.458775 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:35.459142 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 03:10:35.459360 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 03:10:35.459766 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":163,"threads_started":1}
I20260929 03:10:35.462142 24945 heartbeater.cc:347] Connected to a master server at 127.20.174.126:32991
I20260929 03:10:35.462479 24945 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:35.463138 24945 heartbeater.cc:511] Master 127.20.174.126:32991 requested a full tablet report, sending...
I20260929 03:10:35.464952 24749 ts_manager.cc:194] Registered new tserver with Master: d31ca7e4979e4bf39a9fa86fb7c57bf3 (127.20.174.65:45165)
I20260929 03:10:35.466135 24749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:48121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:10:35.477149 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:32991
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:35.593255 24950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:35.593565 24950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:35.593638 24950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:35.597973 24950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:35.598079 24950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:35.598170 24950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:35.603897 24950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:35.605028 24950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:35.606083 24950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:35.616428 24955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:35.616501 24956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:35.616492 24958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:35.616705 24950 server_base.cc:1034] running on GCE node
I20260929 03:10:35.617333 24950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:35.618108 24950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:35.619320 24950 hybrid_clock.cc:648] HybridClock initialized: now 1790651435619251 us; error 83 us; skew 500 ppm
Sep 29 03:10:35 dist-test-slave-k1dg krb5kdc[24709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651435, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:35.622426 24950 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:10:35.623576 24950 webserver.cc:460] Webserver started at http://127.20.174.66:35263/ using document root <none> and password file <none>
I20260929 03:10:35.624186 24950 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:35.624282 24950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:35.624529 24950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:35.626782 24950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "ab78abe45339477b9291d2aaac5d5bc7"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "34c3ce146a22cce6586ef196c6cd1f0f"
server_key_iv: "8f47b9fba45154a2cd021048ac4a1bea"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.627396 24950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "ab78abe45339477b9291d2aaac5d5bc7"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "34c3ce146a22cce6586ef196c6cd1f0f"
server_key_iv: "8f47b9fba45154a2cd021048ac4a1bea"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.631479 24950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:10:35.634148 24965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:35.635566 24950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:10:35.635725 24950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "ab78abe45339477b9291d2aaac5d5bc7"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "34c3ce146a22cce6586ef196c6cd1f0f"
server_key_iv: "8f47b9fba45154a2cd021048ac4a1bea"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.635854 24950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:35.648406 24950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:35.651643 24950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:35.651860 24950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:35.652765 24950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:35.653719 24950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:35.653795 24950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:35.653867 24950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:35.653920 24950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:35.663350 24950 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:36463
I20260929 03:10:35.663368 25078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:36463 every 8 connection(s)
I20260929 03:10:35.664314 24950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:10:35 dist-test-slave-k1dg krb5kdc[24709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651435, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:35.674633 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 24950
I20260929 03:10:35.674746 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:35.675061 21177 external_mini_cluster.cc:1428] Setting key 1ee9e43e4008e6cc7244dbbcece73525
I20260929 03:10:35.680374 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:35.666231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:56361 (local address 127.20.174.126:32991)
0929 03:10:35.666379 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:35.666383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:35.667364 (+ 981us) server_negotiation.cc:407] Connection header received
0929 03:10:35.668849 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:35.668853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:35.668915 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:35.669031 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:35.670266 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.670875 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.672053 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.672277 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.672389 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:35.675213 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:35.675217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:35.675219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:35.675247 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:35.676957 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.677514 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.677520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.677525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.677626 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.678173 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.678179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.678183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.678486 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:35.678609 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:35.679982 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0929 03:10:35.680137 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:10:35.680944 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:35.666745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:32991 (local address 127.20.174.66:56361)
0929 03:10:35.667204 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:35.667248 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:35.668613 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:35.669211 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:35.669218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:35.669598 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:35.670104 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:35.670118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.671006 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.671011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:35.671907 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:35.671918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.672441 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.672446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:35.672623 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:35.673294 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:35.673311 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:35.674981 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:35.677084 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:35.677089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:35.677103 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:35.677386 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:35.677763 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:35.677768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:35.677772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:35.678011 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:35.678620 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:35.678630 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:35.679071 (+ 441us) client_negotiation.cc:769] Sending connection context
0929 03:10:35.679292 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 03:10:35.679697 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260929 03:10:35.682749 25079 heartbeater.cc:347] Connected to a master server at 127.20.174.126:32991
I20260929 03:10:35.683027 25079 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:35.683698 25079 heartbeater.cc:511] Master 127.20.174.126:32991 requested a full tablet report, sending...
I20260929 03:10:35.685055 24749 ts_manager.cc:194] Registered new tserver with Master: ab78abe45339477b9291d2aaac5d5bc7 (127.20.174.66:36463)
I20260929 03:10:35.685916 24749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:56361
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:10:35.709613 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:32991
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:35.827838 25084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:35.828164 25084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:35.828241 25084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:35.832515 25084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:35.832641 25084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:35.832744 25084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:35.838464 25084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:35.839686 25084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:35.840967 25084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:35.851415 25092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:35.851415 25089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:35.851444 25090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:35.851907 25084 server_base.cc:1034] running on GCE node
I20260929 03:10:35.852411 25084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:35.853227 25084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:35.854509 25084 hybrid_clock.cc:648] HybridClock initialized: now 1790651435854487 us; error 37 us; skew 500 ppm
Sep 29 03:10:35 dist-test-slave-k1dg krb5kdc[24709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651435, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:35.857851 25084 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:10:35.858946 25084 webserver.cc:460] Webserver started at http://127.20.174.67:35199/ using document root <none> and password file <none>
I20260929 03:10:35.859530 25084 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:35.859597 25084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:35.859781 25084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:35.862154 25084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "fd2af48211a44fffadeac48911e6c77b"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "0c741c6a83c5ca88b26e2e68e071fc74"
server_key_iv: "a2d0d4fb8140dda287654a029a216250"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.862866 25084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "fd2af48211a44fffadeac48911e6c77b"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "0c741c6a83c5ca88b26e2e68e071fc74"
server_key_iv: "a2d0d4fb8140dda287654a029a216250"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.867233 25084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:10:35.870076 25099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:35.871533 25084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:35.871691 25084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "fd2af48211a44fffadeac48911e6c77b"
format_stamp: "Formatted at 2026-09-29 03:10:35 on dist-test-slave-k1dg"
server_key: "0c741c6a83c5ca88b26e2e68e071fc74"
server_key_iv: "a2d0d4fb8140dda287654a029a216250"
server_key_version: "encryptionkey@0"
I20260929 03:10:35.871814 25084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:35.893718 25084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:35.897408 25084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:35.897616 25084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:35.898242 25084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:35.899200 25084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:35.899256 25084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:35.899324 25084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:35.899359 25084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:35.909696 25084 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:41087
I20260929 03:10:35.909713 25212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:41087 every 8 connection(s)
I20260929 03:10:35.910739 25084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:35.917297 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 25084
I20260929 03:10:35.917459 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:35.917795 21177 external_mini_cluster.cc:1428] Setting key 265e3640a9efe0a298440442ca5bd65e
Sep 29 03:10:35 dist-test-slave-k1dg krb5kdc[24709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651435, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:35.926192 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:35.912787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:33873 (local address 127.20.174.126:32991)
0929 03:10:35.912985 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:35.912989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:35.913850 (+ 861us) server_negotiation.cc:407] Connection header received
0929 03:10:35.915640 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:35.915643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:35.915694 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:35.915773 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:35.917110 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.917975 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.919662 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.919986 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.920102 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:35.922611 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:35.922614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:35.922616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:35.922643 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:35.924182 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.924737 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.924739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.924740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.924815 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.925181 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.925184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.925185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.925361 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:35.925434 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:35.925958 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 03:10:35.926065 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 03:10:35.927161 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:35.913179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:32991 (local address 127.20.174.67:33873)
0929 03:10:35.913672 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:35.913719 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:35.915361 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:35.915999 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:35.916007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:35.916396 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:35.916923 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:35.916938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.918116 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.918119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:35.919505 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:35.919519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.920158 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.920162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:35.920333 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:35.920897 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:35.920914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:35.922393 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:35.924312 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:35.924317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:35.924330 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:35.924634 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:35.924920 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:35.924923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:35.924924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:35.925083 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:35.925448 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:35.925454 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:35.925702 (+ 248us) client_negotiation.cc:769] Sending connection context
0929 03:10:35.925860 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 03:10:35.926180 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":113,"threads_started":1}
I20260929 03:10:35.928407 25213 heartbeater.cc:347] Connected to a master server at 127.20.174.126:32991
I20260929 03:10:35.928663 25213 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:35.929164 25213 heartbeater.cc:511] Master 127.20.174.126:32991 requested a full tablet report, sending...
I20260929 03:10:35.930253 24749 ts_manager.cc:194] Registered new tserver with Master: fd2af48211a44fffadeac48911e6c77b (127.20.174.67:41087)
I20260929 03:10:35.930985 24749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:33873
I20260929 03:10:35.932296 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:10:35 dist-test-slave-k1dg krb5kdc[24709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:10:35 dist-test-slave-k1dg krb5kdc[24709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:10:35.959487 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:35.949206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.20.174.65:45165)
0929 03:10:35.949510 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:35.949524 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:35.949548 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:10:35.949677 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:35.949682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:35.949881 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:35.950017 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:35.950742 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.951538 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.952377 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.952788 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.952875 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:35.955502 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:35.955512 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:35.955525 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:35.955551 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:35.957343 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.957803 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.957809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.957810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.957893 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.958281 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.958284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.958285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.958645 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:35.958845 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:35.959070 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 03:10:35.959288 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
Sep 29 03:10:35 dist-test-slave-k1dg krb5kdc[24709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:35.971643 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:35.962751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.20.174.126:32991)
0929 03:10:35.962879 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:35.962883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:35.963078 (+ 195us) server_negotiation.cc:407] Connection header received
0929 03:10:35.963168 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:35.963170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:35.963209 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:35.963277 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:35.963870 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.964404 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.965251 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:35.965503 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:35.965584 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:35.968280 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:35.968284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:35.968286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:35.968311 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:35.969819 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.970274 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.970278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.970280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.970363 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:35.970703 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:35.970707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:35.970708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:35.970953 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:35.971096 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:35.971341 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:10:35.971473 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 03:10:35.972139 24749 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47290
I20260929 03:10:35.972812 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 24816
I20260929 03:10:35.979033 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 24950
I20260929 03:10:35.984728 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 25084
I20260929 03:10:35.990749 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 24722
2026-09-29T03:10:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2318 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:36 dist-test-slave-k1dg krb5kdc[25225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:36 dist-test-slave-k1dg krb5kdc[25225](info): set up 2 sockets
Sep 29 03:10:36 dist-test-slave-k1dg krb5kdc[25225](info): commencing operation
krb5kdc: starting...
W20260929 03:10:37.030663 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:37 dist-test-slave-k1dg krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:37.158151 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:38805
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:45539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:38805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:37.273211 25238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:37.273577 25238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:37.273662 25238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:37.278142 25238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:37.278271 25238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:37.278307 25238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:37.278350 25238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:37.278386 25238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:37.284195 25238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:45539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:38805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:38805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:37.285406 25238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:37.286643 25238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:37.295531 25246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:37.295527 25244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:37.295573 25243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:37.295989 25238 server_base.cc:1034] running on GCE node
I20260929 03:10:37.296563 25238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:37.297711 25238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:37.298874 25238 hybrid_clock.cc:648] HybridClock initialized: now 1790651437298854 us; error 36 us; skew 500 ppm
Sep 29 03:10:37 dist-test-slave-k1dg krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651437, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:37.307883 25238 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:37.308795 25238 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:37.308902 25238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:37.309154 25238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:37.311488 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "13930b74db524b968d1077b20a7f87f7"
format_stamp: "Formatted at 2026-09-29 03:10:37 on dist-test-slave-k1dg"
server_key: "779774f066c0d23a9de5fd1ec04a938c"
server_key_iv: "545163307cf8e063ea9635a346aada47"
server_key_version: "encryptionkey@0"
I20260929 03:10:37.312144 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "13930b74db524b968d1077b20a7f87f7"
format_stamp: "Formatted at 2026-09-29 03:10:37 on dist-test-slave-k1dg"
server_key: "779774f066c0d23a9de5fd1ec04a938c"
server_key_iv: "545163307cf8e063ea9635a346aada47"
server_key_version: "encryptionkey@0"
I20260929 03:10:37.316229 25238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 03:10:37.318863 25252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:37.320313 25238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:37.320478 25238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "13930b74db524b968d1077b20a7f87f7"
format_stamp: "Formatted at 2026-09-29 03:10:37 on dist-test-slave-k1dg"
server_key: "779774f066c0d23a9de5fd1ec04a938c"
server_key_iv: "545163307cf8e063ea9635a346aada47"
server_key_version: "encryptionkey@0"
I20260929 03:10:37.320613 25238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:37.350822 25238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:37.354581 25238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:37.354825 25238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:37.362322 25238 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:38805
I20260929 03:10:37.362656 25304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:38805 every 8 connection(s)
I20260929 03:10:37.363135 25238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:37.365392 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 25238
I20260929 03:10:37.365525 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:37.365787 21177 external_mini_cluster.cc:1428] Setting key 5dbd5eda4ceaf810b7cfd734ea60b9a6
I20260929 03:10:37.366633 25305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:37.373855 25305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7: Bootstrap starting.
Sep 29 03:10:37 dist-test-slave-k1dg krb5kdc[25225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:37.376713 25305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:37.377519 25305 log.cc:824] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:37.379719 25305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7: No bootstrap required, opened a new log
I20260929 03:10:37.381647 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:37.367269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.20.174.126:38805)
0929 03:10:37.367735 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:37.367762 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:37.367800 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:10:37.368504 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:37.368522 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:37.368871 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:37.369169 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:37.370212 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:37.371437 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:37.372732 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:37.373286 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:37.373425 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:37.376434 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:37.376439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:37.376454 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:37.376485 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:37.378704 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:37.379189 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:37.379194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:37.379198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:37.379330 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:37.379641 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:37.379649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:37.379650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:37.380110 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:37.380325 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:37.380616 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 03:10:37.380915 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":160,"threads_started":1}
I20260929 03:10:37.383924 25305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13930b74db524b968d1077b20a7f87f7" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38805 } }
I20260929 03:10:37.384236 25305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:37.384310 25305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13930b74db524b968d1077b20a7f87f7, State: Initialized, Role: FOLLOWER
I20260929 03:10:37.384688 25305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13930b74db524b968d1077b20a7f87f7" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38805 } }
I20260929 03:10:37.384806 25305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:37.384861 25305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:37.384931 25305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:37.386117 25305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13930b74db524b968d1077b20a7f87f7" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38805 } }
I20260929 03:10:37.386606 25305 leader_election.cc:302] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13930b74db524b968d1077b20a7f87f7; no voters:
I20260929 03:10:37.386943 25305 leader_election.cc:288] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:37.387140 25310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:37.387423 25310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [term 1 LEADER]: Becoming Leader. State: Replica: 13930b74db524b968d1077b20a7f87f7, State: Running, Role: LEADER
I20260929 03:10:37.387758 25310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13930b74db524b968d1077b20a7f87f7" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38805 } }
I20260929 03:10:37.388212 25305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:37.389430 25311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13930b74db524b968d1077b20a7f87f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13930b74db524b968d1077b20a7f87f7" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38805 } } }
I20260929 03:10:37.389565 25311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:37.389875 25312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13930b74db524b968d1077b20a7f87f7. Latest consensus state: current_term: 1 leader_uuid: "13930b74db524b968d1077b20a7f87f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13930b74db524b968d1077b20a7f87f7" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38805 } } }
I20260929 03:10:37.389986 25312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7 [sys.catalog]: This master's current role is: LEADER
W20260929 03:10:37.392874 25325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:10:37.392992 25325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:10:37.393095 25319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:37.394069 25319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:37.400228 25319 catalog_manager.cc:1347] Generated new cluster ID: b67b8582ea104f3a9b92963136574298
I20260929 03:10:37.400408 25319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:37.428483 25319 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:37.429924 25319 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:37.446918 25319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7: Generated new TSK 0
I20260929 03:10:37.447814 25319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:10:37.477632 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38805
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:45539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:37.600085 25331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:37.600397 25331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:37.600485 25331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:37.604846 25331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:37.604943 25331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:37.605033 25331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:37.610859 25331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:45539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:37.612025 25331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:37.613133 25331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:37.622956 25339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:37.622961 25337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:37.622989 25336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:37.623366 25331 server_base.cc:1034] running on GCE node
I20260929 03:10:37.623854 25331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:37.624598 25331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:37.625790 25331 hybrid_clock.cc:648] HybridClock initialized: now 1790651437625772 us; error 30 us; skew 500 ppm
Sep 29 03:10:37 dist-test-slave-k1dg krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651437, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:37.628773 25331 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:10:37.629554 25331 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:37.629652 25331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:37.629905 25331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:37.632155 25331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "64a210f0a8a14b2d8fa860886a23dc82"
format_stamp: "Formatted at 2026-09-29 03:10:37 on dist-test-slave-k1dg"
server_key: "7c77a637744d27d86d864cd0912290ce"
server_key_iv: "be003e792b930353696a1ce65dbea4a2"
server_key_version: "encryptionkey@0"
I20260929 03:10:37.632742 25331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "64a210f0a8a14b2d8fa860886a23dc82"
format_stamp: "Formatted at 2026-09-29 03:10:37 on dist-test-slave-k1dg"
server_key: "7c77a637744d27d86d864cd0912290ce"
server_key_iv: "be003e792b930353696a1ce65dbea4a2"
server_key_version: "encryptionkey@0"
I20260929 03:10:37.637001 25331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:37.639719 25345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:37.641032 25331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:10:37.641187 25331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "64a210f0a8a14b2d8fa860886a23dc82"
format_stamp: "Formatted at 2026-09-29 03:10:37 on dist-test-slave-k1dg"
server_key: "7c77a637744d27d86d864cd0912290ce"
server_key_iv: "be003e792b930353696a1ce65dbea4a2"
server_key_version: "encryptionkey@0"
I20260929 03:10:37.641315 25331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:37.661880 25331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:37.665122 25331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:37.665333 25331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:37.665946 25331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:37.666846 25331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:37.666937 25331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:37.667016 25331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:37.667069 25331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:37.676451 25331 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:38199
I20260929 03:10:37.676488 25458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:38199 every 8 connection(s)
I20260929 03:10:37.677089 25331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:37.685294 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 25331
I20260929 03:10:37.685540 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:37.685956 21177 external_mini_cluster.cc:1428] Setting key 565d8c1d5e670df247ac66fabb08bae4
Sep 29 03:10:37 dist-test-slave-k1dg krb5kdc[25225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651437, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:10:37.700548 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:37.686418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:50083 (local address 127.20.174.126:38805)
0929 03:10:37.686627 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:37.686634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:37.686657 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:10:37.687075 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:37.687079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:37.687143 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:37.687207 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:37.689316 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:37.690274 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:37.692447 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:37.692690 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:37.692813 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:37.696369 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:37.696372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:37.696374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:37.696401 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:37.698027 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:37.698770 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:37.698775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:37.698776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:37.698930 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:37.699292 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:37.699295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:37.699296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:37.699561 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:37.699648 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:37.700235 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 03:10:37.700390 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 03:10:37.701478 25461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:37.679645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:38805 (local address 127.20.174.65:50083)
0929 03:10:37.683457 (+ 3812us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:37.683539 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:37.686781 (+ 3242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:37.687685 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:37.687766 (+ 81us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:37.688305 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:37.689108 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:37.689137 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:37.690447 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:37.690450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:37.692256 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:37.692266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:37.692868 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:37.692872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:37.693086 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:37.694306 (+ 1220us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:37.694328 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:37.696175 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:37.698154 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:37.698159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:37.698173 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:37.698610 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:37.699025 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:37.699027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:37.699029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:37.699171 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:37.699666 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:37.699672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:37.699999 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 03:10:37.700163 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 03:10:37.700552 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2942,"thread_start_us":99,"threads_started":1}
I20260929 03:10:37.703151 25459 heartbeater.cc:347] Connected to a master server at 127.20.174.126:38805
I20260929 03:10:37.703456 25459 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:37.703976 25459 heartbeater.cc:511] Master 127.20.174.126:38805 requested a full tablet report, sending...
I20260929 03:10:37.705798 25269 ts_manager.cc:194] Registered new tserver with Master: 64a210f0a8a14b2d8fa860886a23dc82 (127.20.174.65:38199)
I20260929 03:10:37.707304 25269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:50083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:10:37.718720 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38805
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:45539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:37.836565 25464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:37.836840 25464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:37.836887 25464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:37.840968 25464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:37.841030 25464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:37.841091 25464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:37.847339 25464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:45539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:37.848428 25464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:37.849437 25464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:37.858935 25472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:37.858950 25470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:37.858933 25469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:37.859372 25464 server_base.cc:1034] running on GCE node
I20260929 03:10:37.859792 25464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:37.860460 25464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:37.861636 25464 hybrid_clock.cc:648] HybridClock initialized: now 1790651437861616 us; error 32 us; skew 500 ppm
Sep 29 03:10:37 dist-test-slave-k1dg krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651437, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:37.864670 25464 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:10:37.865492 25464 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:37.865559 25464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:37.865736 25464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:37.867973 25464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "d4724726351b4e39892ea4194cb25e52"
format_stamp: "Formatted at 2026-09-29 03:10:37 on dist-test-slave-k1dg"
server_key: "0fd88d783de28e26016c4ecd823a0ff0"
server_key_iv: "7054e514978aad76272c2cfa4fe8d222"
server_key_version: "encryptionkey@0"
I20260929 03:10:37.868534 25464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "d4724726351b4e39892ea4194cb25e52"
format_stamp: "Formatted at 2026-09-29 03:10:37 on dist-test-slave-k1dg"
server_key: "0fd88d783de28e26016c4ecd823a0ff0"
server_key_iv: "7054e514978aad76272c2cfa4fe8d222"
server_key_version: "encryptionkey@0"
I20260929 03:10:37.872458 25464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:37.875093 25478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:37.876430 25464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:37.876538 25464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "d4724726351b4e39892ea4194cb25e52"
format_stamp: "Formatted at 2026-09-29 03:10:37 on dist-test-slave-k1dg"
server_key: "0fd88d783de28e26016c4ecd823a0ff0"
server_key_iv: "7054e514978aad76272c2cfa4fe8d222"
server_key_version: "encryptionkey@0"
I20260929 03:10:37.876634 25464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:37.926047 25464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:37.929608 25464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:37.929768 25464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:37.930322 25464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:37.931175 25464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:37.931226 25464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:37.931269 25464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:37.931289 25464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:37.940389 25464 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:37957
I20260929 03:10:37.940407 25591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:37957 every 8 connection(s)
I20260929 03:10:37.941061 25464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:37.946244 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 25464
I20260929 03:10:37.946352 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:37.946732 21177 external_mini_cluster.cc:1428] Setting key 25f2a75217c8a40c2b4664e7a81025da
Sep 29 03:10:37 dist-test-slave-k1dg krb5kdc[25225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651437, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:10:37.958243 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:37.943011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:46629 (local address 127.20.174.126:38805)
0929 03:10:37.943150 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:37.943154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:37.944426 (+ 1272us) server_negotiation.cc:407] Connection header received
0929 03:10:37.945539 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:37.945542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:37.945593 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:37.945672 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:37.947985 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:37.948729 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:37.950622 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:37.951040 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:37.951177 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:37.954356 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:37.954360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:37.954362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:37.954390 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:37.955986 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:37.956663 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:37.956665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:37.956667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:37.956742 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:37.957133 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:37.957135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:37.957136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:37.957396 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:37.957483 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:37.958015 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 03:10:37.958119 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:10:37.959170 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:37.943336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:38805 (local address 127.20.174.66:46629)
0929 03:10:37.944250 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:37.944304 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:37.945329 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:37.945970 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:37.946519 (+ 549us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:37.947097 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:37.947810 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:37.947828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:37.948885 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:37.948892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:37.950481 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:37.950501 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:37.951214 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:37.951217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:37.951491 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:37.952140 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:37.952167 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:37.954130 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:37.956120 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:37.956126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:37.956145 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:37.956554 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:37.956839 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:37.956845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:37.956848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:37.957033 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:37.957508 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:37.957516 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:37.957788 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 03:10:37.957960 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 03:10:37.958224 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:10:37.960486 25592 heartbeater.cc:347] Connected to a master server at 127.20.174.126:38805
I20260929 03:10:37.960762 25592 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:37.961265 25592 heartbeater.cc:511] Master 127.20.174.126:38805 requested a full tablet report, sending...
I20260929 03:10:37.962473 25269 ts_manager.cc:194] Registered new tserver with Master: d4724726351b4e39892ea4194cb25e52 (127.20.174.66:37957)
I20260929 03:10:37.963335 25269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:46629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:10:37.979142 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38805
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:45539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:38.094537 25597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:38.094847 25597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:38.094924 25597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:38.099058 25597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:38.099165 25597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:38.099263 25597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:38.104862 25597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:45539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:38.106001 25597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:38.107216 25597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:38.117112 25605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:38.117077 25602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:38.117077 25603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:38.117470 25597 server_base.cc:1034] running on GCE node
I20260929 03:10:38.117926 25597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:38.118671 25597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:38.119843 25597 hybrid_clock.cc:648] HybridClock initialized: now 1790651438119806 us; error 62 us; skew 500 ppm
Sep 29 03:10:38 dist-test-slave-k1dg krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651438, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:38.122757 25597 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:10:38.123582 25597 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:38.123646 25597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:38.123859 25597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:38.126072 25597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "10235903ba734ed88359659fd2c1724a"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "44d97b2e2624b58255c885d64824f995"
server_key_iv: "8765b56a647e2ffb59e8db1b19a6f7ea"
server_key_version: "encryptionkey@0"
I20260929 03:10:38.126674 25597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "10235903ba734ed88359659fd2c1724a"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "44d97b2e2624b58255c885d64824f995"
server_key_iv: "8765b56a647e2ffb59e8db1b19a6f7ea"
server_key_version: "encryptionkey@0"
I20260929 03:10:38.130345 25597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:38.132931 25611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:38.134730 25597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:38.134866 25597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "10235903ba734ed88359659fd2c1724a"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "44d97b2e2624b58255c885d64824f995"
server_key_iv: "8765b56a647e2ffb59e8db1b19a6f7ea"
server_key_version: "encryptionkey@0"
I20260929 03:10:38.135035 25597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:38.150166 25597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:38.153239 25597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:38.153462 25597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:38.154074 25597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:38.155025 25597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:38.155093 25597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:38.155166 25597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:38.155217 25597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:38.164707 25597 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:36991
I20260929 03:10:38.164724 25724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:36991 every 8 connection(s)
I20260929 03:10:38.165323 25597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:38.166203 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 25597
I20260929 03:10:38.166337 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:38.166644 21177 external_mini_cluster.cc:1428] Setting key 6ef351040c0e9fa87fe2affc620ed3bf
Sep 29 03:10:38 dist-test-slave-k1dg krb5kdc[25225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651438, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:38.180397 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.167274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:43387 (local address 127.20.174.126:38805)
0929 03:10:38.167422 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:38.167427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:38.169082 (+ 1655us) server_negotiation.cc:407] Connection header received
0929 03:10:38.170034 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:38.170038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:38.170095 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:38.170168 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:38.171408 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.171942 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.173176 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.173410 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.173535 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.176279 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:38.176282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:38.176284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:38.176311 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:38.177751 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:38.178255 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:38.178258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:38.178259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:38.178349 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:38.178795 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:38.178799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:38.178800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:38.179057 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:38.179151 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:38.180102 (+ 951us) server_negotiation.cc:299] Negotiation successful
0929 03:10:38.180256 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:10:38.181565 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.167846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:38805 (local address 127.20.174.67:43387)
0929 03:10:38.168906 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:38.168968 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:38.169797 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:38.170351 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:38.170359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:38.170748 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:38.171246 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.171260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.172090 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.172094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.173027 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.173038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.173584 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.173589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.173778 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.174524 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:38.174541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:38.176098 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:38.177870 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:38.177875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:38.177888 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:38.178163 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:38.178508 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:38.178510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:38.178512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:38.178659 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:38.179236 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:38.179244 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:38.179613 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 03:10:38.179971 (+ 358us) client_negotiation.cc:240] Negotiation successful
0929 03:10:38.180405 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":140,"threads_started":1}
I20260929 03:10:38.182798 25725 heartbeater.cc:347] Connected to a master server at 127.20.174.126:38805
I20260929 03:10:38.183084 25725 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:38.183562 25725 heartbeater.cc:511] Master 127.20.174.126:38805 requested a full tablet report, sending...
I20260929 03:10:38.184631 25269 ts_manager.cc:194] Registered new tserver with Master: 10235903ba734ed88359659fd2c1724a (127.20.174.67:36991)
I20260929 03:10:38.185328 25269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:43387
I20260929 03:10:38.193275 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:10:38.204051 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.195511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.20.174.126:38805)
0929 03:10:38.195675 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:38.195681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:38.195823 (+ 142us) server_negotiation.cc:407] Connection header received
0929 03:10:38.196011 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:38.196014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:38.196070 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:38.196161 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:38.196920 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.197461 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.198357 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.198663 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.198797 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.200383 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:38.200386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:38.200388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:38.200415 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:38.202023 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:38.202588 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:38.202593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:38.202595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:38.202690 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:38.203090 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:38.203093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:38.203095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:38.203338 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:38.203490 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:38.203754 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 03:10:38.203883 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:10:38.207350 25269 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:38.210657 25269 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:38.227046 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.221111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:37957 (local address 127.0.0.1:37506)
0929 03:10:38.221430 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:38.221456 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:38.221534 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:38.222340 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:38.222342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:38.222359 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:38.222481 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.222488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.223461 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.223465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.225032 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.225042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.226554 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.226557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.226623 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.226640 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 03:10:38.226698 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:10:38.226880 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":307,"thread_start_us":90,"threads_started":1}
I20260929 03:10:38.227248 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.221518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.20.174.66:37957)
0929 03:10:38.221924 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:38.221936 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:38.221955 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:10:38.222029 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:38.222036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:38.222209 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:38.222360 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:38.222568 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.223278 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.225202 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.226112 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.226239 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.226386 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:38.226809 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 03:10:38.227080 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":118,"threads_started":1}
I20260929 03:10:38.227957 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.220215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36628 (local address 127.20.174.65:38199)
0929 03:10:38.220813 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:38.220820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:38.221276 (+ 456us) server_negotiation.cc:407] Connection header received
0929 03:10:38.221357 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:38.221362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:38.221560 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:38.221712 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:38.222128 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.223256 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.225307 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.226112 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.226228 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.226387 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:38.227754 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0929 03:10:38.227849 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":69,"threads_started":1}
I20260929 03:10:38.227662 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.220044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:38199 (local address 127.0.0.1:36628)
0929 03:10:38.221080 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:38.221092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:38.221176 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:38.221860 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:38.221863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:38.221880 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:38.221997 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.222003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.223408 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.223411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.225162 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.225173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.226234 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.226237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.226317 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.226348 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:10:38.227463 (+ 1115us) client_negotiation.cc:240] Negotiation successful
0929 03:10:38.227544 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":254,"thread_start_us":123,"threads_started":1}
I20260929 03:10:38.227845 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.220008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:36991 (local address 127.0.0.1:34818)
0929 03:10:38.220651 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:38.220701 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:38.220828 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:38.221544 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:38.221550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:38.221577 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:38.221737 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.221743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.223499 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.223502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.225162 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.225173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.227512 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.227517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.227615 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.227634 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 03:10:38.227676 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 03:10:38.227748 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":169,"thread_start_us":95,"threads_started":1}
I20260929 03:10:38.228564 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.220314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.20.174.67:36991)
0929 03:10:38.220991 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:38.221003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:38.221026 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:10:38.221106 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:38.221114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:38.221363 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:38.221860 (+ 497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:38.221979 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.223221 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.226488 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.227359 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.227930 (+ 571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.228096 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:38.228158 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:10:38.228258 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":162,"threads_started":1}
I20260929 03:10:38.231132 25393 tablet_service.cc:1467] Processing CreateTablet for tablet c32eb41bdcb444cc8c29f69ea5c36674 (DEFAULT_TABLE table=test-table [id=499b632a6f494ef89cc5651eb37dc964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:38.231140 25659 tablet_service.cc:1467] Processing CreateTablet for tablet c32eb41bdcb444cc8c29f69ea5c36674 (DEFAULT_TABLE table=test-table [id=499b632a6f494ef89cc5651eb37dc964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:38.231145 25526 tablet_service.cc:1467] Processing CreateTablet for tablet c32eb41bdcb444cc8c29f69ea5c36674 (DEFAULT_TABLE table=test-table [id=499b632a6f494ef89cc5651eb37dc964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:38.232184 25393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32eb41bdcb444cc8c29f69ea5c36674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:38.232219 25659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32eb41bdcb444cc8c29f69ea5c36674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:38.232213 25526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32eb41bdcb444cc8c29f69ea5c36674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:38.238420 25742 tablet_bootstrap.cc:492] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82: Bootstrap starting.
I20260929 03:10:38.240041 25744 tablet_bootstrap.cc:492] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52: Bootstrap starting.
I20260929 03:10:38.240935 25742 tablet_bootstrap.cc:654] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:38.242050 25744 tablet_bootstrap.cc:654] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:38.242077 25742 log.cc:824] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:38.242854 25743 tablet_bootstrap.cc:492] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a: Bootstrap starting.
I20260929 03:10:38.243290 25744 log.cc:824] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:38.244148 25742 tablet_bootstrap.cc:492] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82: No bootstrap required, opened a new log
I20260929 03:10:38.244356 25742 ts_tablet_manager.cc:1397] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:10:38.245110 25743 tablet_bootstrap.cc:654] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:38.245407 25744 tablet_bootstrap.cc:492] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52: No bootstrap required, opened a new log
I20260929 03:10:38.245622 25744 ts_tablet_manager.cc:1397] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:10:38.246219 25743 log.cc:824] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:38.248479 25743 tablet_bootstrap.cc:492] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a: No bootstrap required, opened a new log
I20260929 03:10:38.248713 25743 ts_tablet_manager.cc:1397] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:10:38.248582 25742 raft_consensus.cc:348] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10235903ba734ed88359659fd2c1724a" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36991 } } peers { permanent_uuid: "64a210f0a8a14b2d8fa860886a23dc82" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 38199 } } peers { permanent_uuid: "d4724726351b4e39892ea4194cb25e52" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 37957 } }
I20260929 03:10:38.248996 25742 raft_consensus.cc:374] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:38.249094 25742 raft_consensus.cc:729] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a210f0a8a14b2d8fa860886a23dc82, State: Initialized, Role: FOLLOWER
I20260929 03:10:38.249562 25742 consensus_queue.cc:260] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10235903ba734ed88359659fd2c1724a" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36991 } } peers { permanent_uuid: "64a210f0a8a14b2d8fa860886a23dc82" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 38199 } } peers { permanent_uuid: "d4724726351b4e39892ea4194cb25e52" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 37957 } }
I20260929 03:10:38.249562 25744 raft_consensus.cc:348] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10235903ba734ed88359659fd2c1724a" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36991 } } peers { permanent_uuid: "64a210f0a8a14b2d8fa860886a23dc82" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 38199 } } peers { permanent_uuid: "d4724726351b4e39892ea4194cb25e52" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 37957 } }
I20260929 03:10:38.249841 25744 raft_consensus.cc:374] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:38.249899 25744 raft_consensus.cc:729] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4724726351b4e39892ea4194cb25e52, State: Initialized, Role: FOLLOWER
I20260929 03:10:38.250339 25744 consensus_queue.cc:260] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10235903ba734ed88359659fd2c1724a" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36991 } } peers { permanent_uuid: "64a210f0a8a14b2d8fa860886a23dc82" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 38199 } } peers { permanent_uuid: "d4724726351b4e39892ea4194cb25e52" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 37957 } }
I20260929 03:10:38.250595 25742 ts_tablet_manager.cc:1428] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 03:10:38.250986 25459 heartbeater.cc:503] Master 127.20.174.126:38805 was elected leader, sending a full tablet report...
I20260929 03:10:38.251017 25592 heartbeater.cc:503] Master 127.20.174.126:38805 was elected leader, sending a full tablet report...
I20260929 03:10:38.252418 25744 ts_tablet_manager.cc:1428] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 03:10:38.252645 25743 raft_consensus.cc:348] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10235903ba734ed88359659fd2c1724a" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36991 } } peers { permanent_uuid: "64a210f0a8a14b2d8fa860886a23dc82" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 38199 } } peers { permanent_uuid: "d4724726351b4e39892ea4194cb25e52" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 37957 } }
I20260929 03:10:38.252941 25743 raft_consensus.cc:374] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:38.253046 25743 raft_consensus.cc:729] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10235903ba734ed88359659fd2c1724a, State: Initialized, Role: FOLLOWER
I20260929 03:10:38.253531 25743 consensus_queue.cc:260] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10235903ba734ed88359659fd2c1724a" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36991 } } peers { permanent_uuid: "64a210f0a8a14b2d8fa860886a23dc82" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 38199 } } peers { permanent_uuid: "d4724726351b4e39892ea4194cb25e52" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 37957 } }
I20260929 03:10:38.254261 25725 heartbeater.cc:503] Master 127.20.174.126:38805 was elected leader, sending a full tablet report...
I20260929 03:10:38.254618 25743 ts_tablet_manager.cc:1428] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:10:38.302537 25749 raft_consensus.cc:482] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:38.302848 25749 raft_consensus.cc:504] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10235903ba734ed88359659fd2c1724a" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36991 } } peers { permanent_uuid: "64a210f0a8a14b2d8fa860886a23dc82" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 38199 } } peers { permanent_uuid: "d4724726351b4e39892ea4194cb25e52" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 37957 } }
I20260929 03:10:38.304116 25749 leader_election.cc:288] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10235903ba734ed88359659fd2c1724a (127.20.174.67:36991), 64a210f0a8a14b2d8fa860886a23dc82 (127.20.174.65:38199)
I20260929 03:10:38.309629 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.304786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:60323 (local address 127.20.174.65:38199)
0929 03:10:38.304965 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:38.304973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:38.304994 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:10:38.305146 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:38.305151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:38.305268 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:38.305374 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:38.305738 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.306549 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.308223 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.308937 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.309044 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.309103 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:38.309391 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 03:10:38.309468 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 03:10:38.309629 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.304896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:50337 (local address 127.20.174.67:36991)
0929 03:10:38.305028 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:38.305035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:38.305709 (+ 674us) server_negotiation.cc:407] Connection header received
0929 03:10:38.305770 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:38.305773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:38.305832 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:38.305927 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:38.306177 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.306840 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.308293 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.308890 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.309037 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.309091 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:38.309391 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 03:10:38.309468 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:10:38.309635 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.304622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:38199 (local address 127.20.174.66:60323)
0929 03:10:38.304829 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:38.304883 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:38.305001 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:38.305410 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:38.305415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:38.305435 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:38.305561 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.305570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.306658 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.306660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.308007 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.308019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.309050 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.309053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.309231 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.309277 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:10:38.309353 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:10:38.309453 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":65}
I20260929 03:10:38.309643 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.304624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:36991 (local address 127.20.174.66:50337)
0929 03:10:38.305560 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:38.305575 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:38.305689 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:38.305942 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:38.305945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:38.305961 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:38.306058 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.306062 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.306942 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.306945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.308117 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:38.308126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.309050 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.309053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:38.309215 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.309271 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 03:10:38.309363 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:10:38.309455 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":400,"thread_start_us":280,"threads_started":1}
I20260929 03:10:38.310411 25679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32eb41bdcb444cc8c29f69ea5c36674" candidate_uuid: "d4724726351b4e39892ea4194cb25e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10235903ba734ed88359659fd2c1724a" is_pre_election: true
I20260929 03:10:38.310472 25413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32eb41bdcb444cc8c29f69ea5c36674" candidate_uuid: "d4724726351b4e39892ea4194cb25e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a210f0a8a14b2d8fa860886a23dc82" is_pre_election: true
I20260929 03:10:38.310794 25679 raft_consensus.cc:2454] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4724726351b4e39892ea4194cb25e52 in term 0.
I20260929 03:10:38.310796 25413 raft_consensus.cc:2454] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4724726351b4e39892ea4194cb25e52 in term 0.
I20260929 03:10:38.311280 25482 leader_election.cc:302] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10235903ba734ed88359659fd2c1724a, d4724726351b4e39892ea4194cb25e52; no voters:
I20260929 03:10:38.311566 25749 raft_consensus.cc:2781] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:38.311646 25749 raft_consensus.cc:482] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:38.311676 25749 raft_consensus.cc:3037] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:38.313227 25749 raft_consensus.cc:504] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10235903ba734ed88359659fd2c1724a" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36991 } } peers { permanent_uuid: "64a210f0a8a14b2d8fa860886a23dc82" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 38199 } } peers { permanent_uuid: "d4724726351b4e39892ea4194cb25e52" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 37957 } }
I20260929 03:10:38.313632 25749 leader_election.cc:288] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [CANDIDATE]: Term 1 election: Requested vote from peers 10235903ba734ed88359659fd2c1724a (127.20.174.67:36991), 64a210f0a8a14b2d8fa860886a23dc82 (127.20.174.65:38199)
I20260929 03:10:38.314043 25413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32eb41bdcb444cc8c29f69ea5c36674" candidate_uuid: "d4724726351b4e39892ea4194cb25e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a210f0a8a14b2d8fa860886a23dc82"
I20260929 03:10:38.314085 25679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32eb41bdcb444cc8c29f69ea5c36674" candidate_uuid: "d4724726351b4e39892ea4194cb25e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10235903ba734ed88359659fd2c1724a"
I20260929 03:10:38.314191 25413 raft_consensus.cc:3037] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:38.314186 25679 raft_consensus.cc:3037] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:38.315373 25413 raft_consensus.cc:2454] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4724726351b4e39892ea4194cb25e52 in term 1.
I20260929 03:10:38.315375 25679 raft_consensus.cc:2454] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4724726351b4e39892ea4194cb25e52 in term 1.
I20260929 03:10:38.315737 25479 leader_election.cc:302] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64a210f0a8a14b2d8fa860886a23dc82, d4724726351b4e39892ea4194cb25e52; no voters:
I20260929 03:10:38.315920 25749 raft_consensus.cc:2781] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:38.316165 25749 raft_consensus.cc:686] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [term 1 LEADER]: Becoming Leader. State: Replica: d4724726351b4e39892ea4194cb25e52, State: Running, Role: LEADER
I20260929 03:10:38.316527 25749 consensus_queue.cc:237] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10235903ba734ed88359659fd2c1724a" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36991 } } peers { permanent_uuid: "64a210f0a8a14b2d8fa860886a23dc82" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 38199 } } peers { permanent_uuid: "d4724726351b4e39892ea4194cb25e52" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 37957 } }
I20260929 03:10:38.319983 25268 catalog_manager.cc:5495] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4724726351b4e39892ea4194cb25e52 (127.20.174.66). New cstate: current_term: 1 leader_uuid: "d4724726351b4e39892ea4194cb25e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10235903ba734ed88359659fd2c1724a" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64a210f0a8a14b2d8fa860886a23dc82" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 38199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4724726351b4e39892ea4194cb25e52" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 37957 } health_report { overall_health: HEALTHY } } }
I20260929 03:10:38.350263 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.346340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.20.174.66:37957)
0929 03:10:38.346546 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:38.346549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:38.346565 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:10:38.346692 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:38.346694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:38.346748 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:38.346843 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:38.347113 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.347626 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.348501 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:38.348713 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:38.348796 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:38.349221 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:38.349646 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:38.349737 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:38.349979 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 03:10:38.350037 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:10:38.356555 25413 raft_consensus.cc:1261] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82 [term 1 FOLLOWER]: Refusing update from remote peer d4724726351b4e39892ea4194cb25e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:38.356555 25679 raft_consensus.cc:1261] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a [term 1 FOLLOWER]: Refusing update from remote peer d4724726351b4e39892ea4194cb25e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:38.357312 25749 consensus_queue.cc:1035] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10235903ba734ed88359659fd2c1724a" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:38.357542 25752 consensus_queue.cc:1035] T c32eb41bdcb444cc8c29f69ea5c36674 P d4724726351b4e39892ea4194cb25e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a210f0a8a14b2d8fa860886a23dc82" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 38199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:38.366962 25755 mvcc.cc:205] Tried to move back new op lower bound from 7334508291500126208 to 7334508291347238912. Current Snapshot: MvccSnapshot[applied={T|T < 7334508291500126208}]
I20260929 03:10:38.368736 25756 mvcc.cc:205] Tried to move back new op lower bound from 7334508291500126208 to 7334508291347238912. Current Snapshot: MvccSnapshot[applied={T|T < 7334508291500126208}]
I20260929 03:10:38.370636 25757 mvcc.cc:205] Tried to move back new op lower bound from 7334508291500126208 to 7334508291347238912. Current Snapshot: MvccSnapshot[applied={T|T < 7334508291500126208}]
I20260929 03:10:38.377022 25269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:38.377205 25269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:38.379850 25269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13930b74db524b968d1077b20a7f87f7: Sending DeleteTablet for 3 replicas of tablet c32eb41bdcb444cc8c29f69ea5c36674
I20260929 03:10:38.380674 25659 tablet_service.cc:1514] Processing DeleteTablet for tablet c32eb41bdcb444cc8c29f69ea5c36674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:38 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:34818
I20260929 03:10:38.380751 25526 tablet_service.cc:1514] Processing DeleteTablet for tablet c32eb41bdcb444cc8c29f69ea5c36674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:38 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:37506
I20260929 03:10:38.380893 25393 tablet_service.cc:1514] Processing DeleteTablet for tablet c32eb41bdcb444cc8c29f69ea5c36674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:38 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:36628
I20260929 03:10:38.381352 25766 tablet_replica.cc:330] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82: stopping tablet replica
I20260929 03:10:38.381381 25764 tablet_replica.cc:330] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a: stopping tablet replica
I20260929 03:10:38.381750 25764 raft_consensus.cc:2229] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:38.381750 25766 raft_consensus.cc:2229] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:38.381876 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 25331
I20260929 03:10:38.382112 25766 raft_consensus.cc:2258] T c32eb41bdcb444cc8c29f69ea5c36674 P 64a210f0a8a14b2d8fa860886a23dc82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:38.383984 25764 raft_consensus.cc:2258] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:38.385785 25764 ts_tablet_manager.cc:1905] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:38.389559 25764 ts_tablet_manager.cc:1918] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:38.389737 25764 log.cc:1196] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/c32eb41bdcb444cc8c29f69ea5c36674
W20260929 03:10:38.389875 25253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:38199 (error 108)
I20260929 03:10:38.390177 25764 ts_tablet_manager.cc:1939] T c32eb41bdcb444cc8c29f69ea5c36674 P 10235903ba734ed88359659fd2c1724a: Deleting consensus metadata
I20260929 03:10:38.391130 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.390781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:38199 (local address 127.0.0.1:36642)
0929 03:10:38.390931 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:38.391038 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:38199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 03:10:38.391165 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 25464
W20260929 03:10:38.391347 25253 catalog_manager.cc:4584] TS 64a210f0a8a14b2d8fa860886a23dc82 (127.20.174.65:38199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c32eb41bdcb444cc8c29f69ea5c36674: Network error: Client connection negotiation failed: client connection to 127.20.174.65:38199: connect: Connection refused (error 111)
I20260929 03:10:38.392369 25256 catalog_manager.cc:4841] TS 10235903ba734ed88359659fd2c1724a (127.20.174.67:36991): tablet c32eb41bdcb444cc8c29f69ea5c36674 (table test-table [id=499b632a6f494ef89cc5651eb37dc964]) successfully deleted
I20260929 03:10:38.396543 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 25597
I20260929 03:10:38.398500 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.396860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:37957 (local address 127.0.0.1:37532)
0929 03:10:38.398271 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:38.398362 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.66:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1282}
I20260929 03:10:38.402637 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 25238
2026-09-29T03:10:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2412 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T03:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:38Z Disabled control of system clock
I20260929 03:10:38.435060 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:36501
--webserver_interface=127.20.174.126
--webserver_port=0
--builtin_ntp_servers=127.20.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:36501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:10:38.549849 25771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:38.550220 25771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:38.550263 25771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:38.554364 25771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:38.554507 25771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:38.554548 25771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:38.554565 25771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:38.554585 25771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:38.559428 25771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:36501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:38.560542 25771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:38.561879 25771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:38.570799 25776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:38.570933 25777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:38.570847 25779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:38.571064 25771 server_base.cc:1034] running on GCE node
I20260929 03:10:38.571696 25771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:38.572872 25771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:38.574112 25771 hybrid_clock.cc:648] HybridClock initialized: now 1790651438574083 us; error 48 us; skew 500 ppm
I20260929 03:10:38.576138 25771 webserver.cc:460] Webserver started at http://127.20.174.126:32989/ using document root <none> and password file <none>
I20260929 03:10:38.576735 25771 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:38.576831 25771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:38.577078 25771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:38.579337 25771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "eba03eb77f794568b186ac5d803edbfb"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "5c39fb191086bd755188040618bd91ed"
server_key_iv: "08dfe7654421772be221008e41848d7a"
server_key_version: "encryptionkey@0"
I20260929 03:10:38.579941 25771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "eba03eb77f794568b186ac5d803edbfb"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "5c39fb191086bd755188040618bd91ed"
server_key_iv: "08dfe7654421772be221008e41848d7a"
server_key_version: "encryptionkey@0"
I20260929 03:10:38.583959 25771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:10:38.586687 25785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:38.588105 25771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:10:38.588261 25771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "eba03eb77f794568b186ac5d803edbfb"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "5c39fb191086bd755188040618bd91ed"
server_key_iv: "08dfe7654421772be221008e41848d7a"
server_key_version: "encryptionkey@0"
I20260929 03:10:38.588392 25771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:38.593305 25771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:38.593935 25771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:38.594128 25771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:38.601195 25771 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:36501
I20260929 03:10:38.601228 25837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:36501 every 8 connection(s)
I20260929 03:10:38.602164 25771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:38.605942 25838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:38.611230 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 25771
I20260929 03:10:38.611387 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:38.611692 21177 external_mini_cluster.cc:1428] Setting key 7613d1333aac975f7ba22e2c3297bbc7
I20260929 03:10:38.613590 25838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb: Bootstrap starting.
I20260929 03:10:38.616320 25838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:38.617298 25838 log.cc:824] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:38.618000 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.613333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.20.174.126:36501)
0929 03:10:38.614123 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:38.614140 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:38.614179 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:10:38.614933 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:38.614960 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:38.615342 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:38.615719 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:38.616621 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:38.616626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:38.616639 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:38.616649 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:38.616706 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:38.616722 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:38.616845 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:38.616963 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 03:10:38.617217 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":171,"threads_started":1}
I20260929 03:10:38.619747 25838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb: No bootstrap required, opened a new log
I20260929 03:10:38.623835 25838 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba03eb77f794568b186ac5d803edbfb" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36501 } }
I20260929 03:10:38.624091 25838 raft_consensus.cc:374] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:38.624167 25838 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba03eb77f794568b186ac5d803edbfb, State: Initialized, Role: FOLLOWER
I20260929 03:10:38.624642 25838 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba03eb77f794568b186ac5d803edbfb" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36501 } }
I20260929 03:10:38.624780 25838 raft_consensus.cc:388] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:38.624873 25838 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:38.624995 25838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:38.626080 25838 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba03eb77f794568b186ac5d803edbfb" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36501 } }
I20260929 03:10:38.626560 25838 leader_election.cc:302] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eba03eb77f794568b186ac5d803edbfb; no voters:
I20260929 03:10:38.626947 25838 leader_election.cc:288] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:38.627137 25843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:38.627375 25843 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [term 1 LEADER]: Becoming Leader. State: Replica: eba03eb77f794568b186ac5d803edbfb, State: Running, Role: LEADER
I20260929 03:10:38.627739 25843 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba03eb77f794568b186ac5d803edbfb" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36501 } }
I20260929 03:10:38.628407 25838 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:38.629529 25845 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba03eb77f794568b186ac5d803edbfb. Latest consensus state: current_term: 1 leader_uuid: "eba03eb77f794568b186ac5d803edbfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba03eb77f794568b186ac5d803edbfb" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36501 } } }
I20260929 03:10:38.629675 25845 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:38.630088 25852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:38.630049 25844 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eba03eb77f794568b186ac5d803edbfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba03eb77f794568b186ac5d803edbfb" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36501 } } }
I20260929 03:10:38.630201 25844 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:38.633245 25852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:38.639050 25852 catalog_manager.cc:1347] Generated new cluster ID: d9f94422968f426ba6e3fbeb0a99ff55
I20260929 03:10:38.639135 25852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:38.649860 25852 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:38.650204 25852 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:38.659991 25852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb: Generated new TSK 0
I20260929 03:10:38.660826 25852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:10:38.674057 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36501
--builtin_ntp_servers=127.20.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:10:38.791572 25862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:38.791926 25862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:38.792042 25862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:38.796495 25862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:38.796597 25862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:38.796689 25862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:38.801977 25862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:38.803226 25862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:38.804694 25862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:38.815358 25870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:38.815332 25867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:38.815332 25868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:38.815737 25862 server_base.cc:1034] running on GCE node
I20260929 03:10:38.816272 25862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:38.817056 25862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:38.818225 25862 hybrid_clock.cc:648] HybridClock initialized: now 1790651438818208 us; error 31 us; skew 500 ppm
I20260929 03:10:38.820236 25862 webserver.cc:460] Webserver started at http://127.20.174.65:37829/ using document root <none> and password file <none>
I20260929 03:10:38.820856 25862 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:38.820922 25862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:38.821192 25862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:38.823428 25862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "d871ab17fd294435ab216d83e61e4c7d"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "2789fb1625e67bdd467027e64df8fb7e"
server_key_iv: "eb8faf16f525a52aa75045ed507be26d"
server_key_version: "encryptionkey@0"
I20260929 03:10:38.824043 25862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "d871ab17fd294435ab216d83e61e4c7d"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "2789fb1625e67bdd467027e64df8fb7e"
server_key_iv: "eb8faf16f525a52aa75045ed507be26d"
server_key_version: "encryptionkey@0"
I20260929 03:10:38.828115 25862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:10:38.831209 25876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:38.832585 25862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:10:38.832691 25862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "d871ab17fd294435ab216d83e61e4c7d"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "2789fb1625e67bdd467027e64df8fb7e"
server_key_iv: "eb8faf16f525a52aa75045ed507be26d"
server_key_version: "encryptionkey@0"
I20260929 03:10:38.832792 25862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:38.837409 25862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:38.838011 25862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:38.838168 25862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:38.838762 25862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:38.839689 25862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:38.839749 25862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:38.839790 25862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:38.839807 25862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:38.848762 25862 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:37577
I20260929 03:10:38.848784 25989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:37577 every 8 connection(s)
I20260929 03:10:38.849787 25862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:38.850872 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 25862
I20260929 03:10:38.850975 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:38.851250 21177 external_mini_cluster.cc:1428] Setting key 0da3d13c0fcc51f76c5a0dcc67d2d154
I20260929 03:10:38.853741 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36501
--builtin_ntp_servers=127.20.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 03:10:38.857889 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.852227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:35409 (local address 127.20.174.126:36501)
0929 03:10:38.852397 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:38.852401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:38.853778 (+ 1377us) server_negotiation.cc:407] Connection header received
0929 03:10:38.855824 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:38.855831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:38.855884 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:38.855998 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:38.857284 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:38.857292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:38.857298 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:38.857318 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:38.857347 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:38.857361 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:38.857475 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:38.857621 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:10:38.857679 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:10:38.858907 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:38.852613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:36501 (local address 127.20.174.65:35409)
0929 03:10:38.853585 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:38.853650 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:38.855569 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:38.856168 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:38.856175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:38.856277 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:38.856961 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:10:38.856962 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:38.857008 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:10:38.857087 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:38.857457 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:38.857464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:38.857468 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:10:38.857652 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 03:10:38.857911 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":203,"threads_started":1}
I20260929 03:10:38.860361 25990 heartbeater.cc:347] Connected to a master server at 127.20.174.126:36501
I20260929 03:10:38.860682 25990 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:38.861302 25990 heartbeater.cc:511] Master 127.20.174.126:36501 requested a full tablet report, sending...
I20260929 03:10:38.863523 25802 ts_manager.cc:194] Registered new tserver with Master: d871ab17fd294435ab216d83e61e4c7d (127.20.174.65:37577)
W20260929 03:10:38.974980 25993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:38.975306 25993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:38.975353 25993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:38.979869 25993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:38.979946 25993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:38.980065 25993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:38.985209 25993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:38.986385 25993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:38.987975 25993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:38.998596 25998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:38.998652 26001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:38.998679 25999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:38.999076 25993 server_base.cc:1034] running on GCE node
I20260929 03:10:38.999545 25993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:39.000276 25993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:39.001483 25993 hybrid_clock.cc:648] HybridClock initialized: now 1790651439001428 us; error 53 us; skew 500 ppm
I20260929 03:10:39.003706 25993 webserver.cc:460] Webserver started at http://127.20.174.66:45873/ using document root <none> and password file <none>
I20260929 03:10:39.004513 25993 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:39.004592 25993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:39.004858 25993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:39.007422 25993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "780f5027552dada4c75723195f286a6c"
server_key_iv: "3b913fa2f6393e57de1e7b2d77c62d88"
server_key_version: "encryptionkey@0"
I20260929 03:10:39.008132 25993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "780f5027552dada4c75723195f286a6c"
server_key_iv: "3b913fa2f6393e57de1e7b2d77c62d88"
server_key_version: "encryptionkey@0"
I20260929 03:10:39.012666 25993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 03:10:39.015745 26007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:39.017329 25993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:39.017510 25993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e"
format_stamp: "Formatted at 2026-09-29 03:10:38 on dist-test-slave-k1dg"
server_key: "780f5027552dada4c75723195f286a6c"
server_key_iv: "3b913fa2f6393e57de1e7b2d77c62d88"
server_key_version: "encryptionkey@0"
I20260929 03:10:39.017666 25993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:39.022784 25993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:39.023473 25993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:39.023682 25993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:39.024271 25993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:39.025262 25993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:39.025338 25993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:39.025409 25993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:39.025456 25993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:39.035105 25993 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:44979
I20260929 03:10:39.035126 26120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:44979 every 8 connection(s)
I20260929 03:10:39.036206 25993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:39.041390 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 25993
I20260929 03:10:39.041569 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:39.041882 21177 external_mini_cluster.cc:1428] Setting key 52257a0d7f07878eed7d093375024046
I20260929 03:10:39.043434 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.038329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:49585 (local address 127.20.174.126:36501)
0929 03:10:39.038515 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:39.038521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:39.039857 (+ 1336us) server_negotiation.cc:407] Connection header received
0929 03:10:39.041212 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:39.041216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:39.041302 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:39.041423 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:39.042871 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:39.042874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:39.042877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:39.042890 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:39.042914 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:39.042924 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:39.043037 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:39.043210 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 03:10:39.043257 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:10:39.044190 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36501
--builtin_ntp_servers=127.20.174.84:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 03:10:39.044556 26123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.038885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:36501 (local address 127.20.174.66:49585)
0929 03:10:39.039629 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:39.039678 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:39.040946 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:39.041703 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:39.041712 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:39.041813 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:39.042527 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:10:39.042529 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:39.042589 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:10:39.042688 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:39.043050 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:39.043058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:39.043064 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:10:39.043229 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 03:10:39.043599 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":122,"threads_started":1}
I20260929 03:10:39.045915 26121 heartbeater.cc:347] Connected to a master server at 127.20.174.126:36501
I20260929 03:10:39.046180 26121 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:39.046726 26121 heartbeater.cc:511] Master 127.20.174.126:36501 requested a full tablet report, sending...
I20260929 03:10:39.047926 25802 ts_manager.cc:194] Registered new tserver with Master: 8c94c62f370b4c0c9f62e1cdba50bd5e (127.20.174.66:44979)
W20260929 03:10:39.162253 26124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:39.162633 26124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:39.162714 26124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:39.167054 26124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:39.167121 26124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:39.167213 26124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:39.172556 26124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:39.173744 26124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:39.175278 26124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:39.186013 26130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:39.186013 26129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:39.186043 26132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:39.186252 26124 server_base.cc:1034] running on GCE node
I20260929 03:10:39.186904 26124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:39.187726 26124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:39.188954 26124 hybrid_clock.cc:648] HybridClock initialized: now 1790651439188931 us; error 35 us; skew 500 ppm
I20260929 03:10:39.191284 26124 webserver.cc:460] Webserver started at http://127.20.174.67:40459/ using document root <none> and password file <none>
I20260929 03:10:39.192000 26124 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:39.192096 26124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:39.192324 26124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:39.194655 26124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "b091a31f41fa4314a6f422e74d7f222b"
format_stamp: "Formatted at 2026-09-29 03:10:39 on dist-test-slave-k1dg"
server_key: "1014569f4fd910b36fc56e25dc5ae71b"
server_key_iv: "0d496483a4ccba496e7c71c32b49cc9c"
server_key_version: "encryptionkey@0"
I20260929 03:10:39.195305 26124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "b091a31f41fa4314a6f422e74d7f222b"
format_stamp: "Formatted at 2026-09-29 03:10:39 on dist-test-slave-k1dg"
server_key: "1014569f4fd910b36fc56e25dc5ae71b"
server_key_iv: "0d496483a4ccba496e7c71c32b49cc9c"
server_key_version: "encryptionkey@0"
I20260929 03:10:39.199419 26124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:10:39.202345 26138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:39.203846 26124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:10:39.204005 26124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "b091a31f41fa4314a6f422e74d7f222b"
format_stamp: "Formatted at 2026-09-29 03:10:39 on dist-test-slave-k1dg"
server_key: "1014569f4fd910b36fc56e25dc5ae71b"
server_key_iv: "0d496483a4ccba496e7c71c32b49cc9c"
server_key_version: "encryptionkey@0"
I20260929 03:10:39.204147 26124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:39.209328 26124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:39.210054 26124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:39.210258 26124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:39.210928 26124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:39.211984 26124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:39.212061 26124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:39.212133 26124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:39.212184 26124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:39.221837 26124 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:45553
I20260929 03:10:39.221855 26251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:45553 every 8 connection(s)
I20260929 03:10:39.222967 26124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:39.229550 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.224899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:42031 (local address 127.20.174.126:36501)
0929 03:10:39.225113 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:39.225118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:39.226228 (+ 1110us) server_negotiation.cc:407] Connection header received
0929 03:10:39.227568 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:39.227571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:39.227621 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:39.227703 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:39.228977 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:39.228980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:39.228982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:39.228995 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:39.229021 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:39.229030 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:39.229216 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:39.229381 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:10:39.229427 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 03:10:39.230612 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.225383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:36501 (local address 127.20.174.67:42031)
0929 03:10:39.226098 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:39.226140 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:39.227342 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:39.227909 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:39.227951 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:39.228049 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:39.228609 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:10:39.228611 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:39.228654 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:10:39.228789 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:39.229203 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:39.229220 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:39.229225 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:10:39.229376 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 03:10:39.229590 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
I20260929 03:10:39.231752 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 26124
I20260929 03:10:39.231889 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:39.231870 26252 heartbeater.cc:347] Connected to a master server at 127.20.174.126:36501
I20260929 03:10:39.232167 26252 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:39.232173 21177 external_mini_cluster.cc:1428] Setting key 3a3e7cb565f33a9945ef440ff670cd31
I20260929 03:10:39.232656 26252 heartbeater.cc:511] Master 127.20.174.126:36501 requested a full tablet report, sending...
I20260929 03:10:39.233841 25801 ts_manager.cc:194] Registered new tserver with Master: b091a31f41fa4314a6f422e74d7f222b (127.20.174.67:45553)
I20260929 03:10:39.246888 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:10:39.250842 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.248970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.20.174.126:36501)
0929 03:10:39.249159 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:39.249164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:39.249316 (+ 152us) server_negotiation.cc:407] Connection header received
0929 03:10:39.249500 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:39.249502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:39.249538 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:39.249624 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:39.250338 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:39.250342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:39.250347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:39.250365 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:39.250385 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:39.250393 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:39.250537 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:39.250633 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:10:39.250681 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:10:39.254194 25801 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:39.257365 25801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:39.270161 26264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.267129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:44979 (local address 127.0.0.1:35758)
0929 03:10:39.267875 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:39.267889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:39.267987 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:39.268576 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:39.268582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:39.268662 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:39.269291 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:10:39.269293 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:39.269316 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:10:39.269395 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:39.269793 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:39.269802 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:39.269808 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:10:39.269924 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 03:10:39.270007 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":9,"thread_start_us":108,"threads_started":1}
I20260929 03:10:39.270174 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.267248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.20.174.66:44979)
0929 03:10:39.267825 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:39.267836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:39.268098 (+ 262us) server_negotiation.cc:407] Connection header received
0929 03:10:39.268169 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:39.268175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:39.268420 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:39.268585 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:39.269572 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:39.269578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:39.269591 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:39.269604 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:39.269658 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:39.269686 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:39.269841 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:39.269938 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:10:39.270016 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":122,"threads_started":1}
I20260929 03:10:39.270900 26263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.267129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:45553 (local address 127.0.0.1:60030)
0929 03:10:39.267497 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:39.267553 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:39.267657 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:39.269158 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:39.269161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:39.269176 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:39.269599 (+ 423us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:10:39.269601 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:39.269612 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:10:39.269638 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:39.270560 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:39.270563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:39.270567 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:10:39.270668 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 03:10:39.270709 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
I20260929 03:10:39.271009 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.268420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60030 (local address 127.20.174.67:45553)
0929 03:10:39.268742 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:39.268753 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:39.268776 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:10:39.268825 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:39.268830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:39.269011 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:39.269413 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:39.270313 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:39.270319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:39.270329 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:39.270339 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:39.270399 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:39.270470 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:39.270589 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:39.270688 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:10:39.270803 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":142,"threads_started":1}
I20260929 03:10:39.271819 26266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.267921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:37577 (local address 127.0.0.1:35700)
0929 03:10:39.268222 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:39.268234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:39.268302 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:39.270400 (+ 2098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:39.270403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:39.270418 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:39.270930 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:10:39.270932 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:39.270943 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:10:39.270968 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:39.271590 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:39.271593 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:39.271596 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:10:39.271664 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:10:39.271698 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":92,"threads_started":1}
I20260929 03:10:39.272439 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.268098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.20.174.65:37577)
0929 03:10:39.269447 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:39.269460 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:39.269484 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:10:39.269534 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:39.269539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:39.269705 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:39.269826 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:39.271120 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:39.271127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:39.271140 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:39.271154 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:39.271198 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:39.271265 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:39.271548 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:39.272221 (+ 673us) server_negotiation.cc:299] Negotiation successful
0929 03:10:39.272296 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243,"thread_start_us":101,"threads_started":1}
I20260929 03:10:39.273939 26186 tablet_service.cc:1467] Processing CreateTablet for tablet 786fce2d898a44d9b8fd4a563caaa10b (DEFAULT_TABLE table=test-table [id=3466c686573d4f8a80c076e4426bbf5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:39.273973 26055 tablet_service.cc:1467] Processing CreateTablet for tablet 786fce2d898a44d9b8fd4a563caaa10b (DEFAULT_TABLE table=test-table [id=3466c686573d4f8a80c076e4426bbf5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:39.275084 26055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786fce2d898a44d9b8fd4a563caaa10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:39.275084 26186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786fce2d898a44d9b8fd4a563caaa10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:39.275226 25924 tablet_service.cc:1467] Processing CreateTablet for tablet 786fce2d898a44d9b8fd4a563caaa10b (DEFAULT_TABLE table=test-table [id=3466c686573d4f8a80c076e4426bbf5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:39.276211 25924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786fce2d898a44d9b8fd4a563caaa10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:39.281400 26269 tablet_bootstrap.cc:492] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: Bootstrap starting.
I20260929 03:10:39.283177 26270 tablet_bootstrap.cc:492] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: Bootstrap starting.
I20260929 03:10:39.284294 26271 tablet_bootstrap.cc:492] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d: Bootstrap starting.
I20260929 03:10:39.284366 26269 tablet_bootstrap.cc:654] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:39.285128 26269 log.cc:824] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:39.285549 26270 tablet_bootstrap.cc:654] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:39.286381 26270 log.cc:824] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:39.286634 26271 tablet_bootstrap.cc:654] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:39.287268 26269 tablet_bootstrap.cc:492] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: No bootstrap required, opened a new log
I20260929 03:10:39.287487 26269 ts_tablet_manager.cc:1397] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:10:39.287465 26271 log.cc:824] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:39.288496 26270 tablet_bootstrap.cc:492] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: No bootstrap required, opened a new log
I20260929 03:10:39.288673 26270 ts_tablet_manager.cc:1397] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:10:39.289459 26271 tablet_bootstrap.cc:492] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d: No bootstrap required, opened a new log
I20260929 03:10:39.289700 26271 ts_tablet_manager.cc:1397] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260929 03:10:39.292086 26269 raft_consensus.cc:348] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44979 } } peers { permanent_uuid: "b091a31f41fa4314a6f422e74d7f222b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45553 } } peers { permanent_uuid: "d871ab17fd294435ab216d83e61e4c7d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 37577 } }
I20260929 03:10:39.292374 26269 raft_consensus.cc:374] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:39.292433 26269 raft_consensus.cc:729] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b091a31f41fa4314a6f422e74d7f222b, State: Initialized, Role: FOLLOWER
I20260929 03:10:39.292882 26269 consensus_queue.cc:260] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44979 } } peers { permanent_uuid: "b091a31f41fa4314a6f422e74d7f222b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45553 } } peers { permanent_uuid: "d871ab17fd294435ab216d83e61e4c7d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 37577 } }
I20260929 03:10:39.293500 26270 raft_consensus.cc:348] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44979 } } peers { permanent_uuid: "b091a31f41fa4314a6f422e74d7f222b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45553 } } peers { permanent_uuid: "d871ab17fd294435ab216d83e61e4c7d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 37577 } }
I20260929 03:10:39.293764 26270 raft_consensus.cc:374] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:39.293830 26270 raft_consensus.cc:729] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c94c62f370b4c0c9f62e1cdba50bd5e, State: Initialized, Role: FOLLOWER
I20260929 03:10:39.293874 26269 ts_tablet_manager.cc:1428] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:10:39.294256 26252 heartbeater.cc:503] Master 127.20.174.126:36501 was elected leader, sending a full tablet report...
I20260929 03:10:39.294401 26270 consensus_queue.cc:260] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44979 } } peers { permanent_uuid: "b091a31f41fa4314a6f422e74d7f222b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45553 } } peers { permanent_uuid: "d871ab17fd294435ab216d83e61e4c7d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 37577 } }
I20260929 03:10:39.294559 26271 raft_consensus.cc:348] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44979 } } peers { permanent_uuid: "b091a31f41fa4314a6f422e74d7f222b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45553 } } peers { permanent_uuid: "d871ab17fd294435ab216d83e61e4c7d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 37577 } }
I20260929 03:10:39.294817 26271 raft_consensus.cc:374] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:39.294878 26271 raft_consensus.cc:729] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871ab17fd294435ab216d83e61e4c7d, State: Initialized, Role: FOLLOWER
I20260929 03:10:39.295230 26121 heartbeater.cc:503] Master 127.20.174.126:36501 was elected leader, sending a full tablet report...
I20260929 03:10:39.295545 26271 consensus_queue.cc:260] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44979 } } peers { permanent_uuid: "b091a31f41fa4314a6f422e74d7f222b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45553 } } peers { permanent_uuid: "d871ab17fd294435ab216d83e61e4c7d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 37577 } }
I20260929 03:10:39.296283 25990 heartbeater.cc:503] Master 127.20.174.126:36501 was elected leader, sending a full tablet report...
I20260929 03:10:39.296559 26271 ts_tablet_manager.cc:1428] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 03:10:39.297497 26270 ts_tablet_manager.cc:1428] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20260929 03:10:39.351171 25991 tablet.cc:2369] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:10:39.474066 26253 tablet.cc:2369] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:10:39.516975 26275 raft_consensus.cc:482] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:39.517198 26275 raft_consensus.cc:504] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44979 } } peers { permanent_uuid: "b091a31f41fa4314a6f422e74d7f222b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45553 } } peers { permanent_uuid: "d871ab17fd294435ab216d83e61e4c7d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 37577 } }
I20260929 03:10:39.518477 26275 leader_election.cc:288] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c94c62f370b4c0c9f62e1cdba50bd5e (127.20.174.66:44979), d871ab17fd294435ab216d83e61e4c7d (127.20.174.65:37577)
I20260929 03:10:39.520737 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.518905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:47513 (local address 127.20.174.66:44979)
0929 03:10:39.519071 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:39.519078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:39.519095 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:10:39.519144 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:39.519147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:39.519184 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:39.519296 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:39.520223 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:39.520227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:39.520230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:39.520250 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:39.520299 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:39.520308 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:39.520418 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:39.520549 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 03:10:39.520591 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 03:10:39.520763 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.519014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:51921 (local address 127.20.174.65:37577)
0929 03:10:39.519175 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:39.519180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:39.519441 (+ 261us) server_negotiation.cc:407] Connection header received
0929 03:10:39.519501 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:39.519507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:39.519551 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:39.519632 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:39.520341 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:39.520344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:39.520346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:39.520358 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:39.520389 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:39.520402 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:39.520498 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:39.520591 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 03:10:39.520630 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:10:39.520742 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.518693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:44979 (local address 127.20.174.67:47513)
0929 03:10:39.518830 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:39.518846 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:39.518936 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:39.519314 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:39.519317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:39.519333 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:39.520021 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:10:39.520023 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:39.520034 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:10:39.520062 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:39.520413 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:39.520417 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:39.520420 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:10:39.520537 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 03:10:39.520580 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 03:10:39.520782 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.518936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:37577 (local address 127.20.174.67:51921)
0929 03:10:39.519338 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:39.519350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:39.519437 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:39.519688 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:39.519691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:39.519709 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:39.520164 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:10:39.520166 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:39.520175 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:10:39.520199 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:39.520492 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:39.520495 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:39.520498 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:10:39.520582 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:10:39.520621 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":178,"threads_started":1}
I20260929 03:10:39.521695 26075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786fce2d898a44d9b8fd4a563caaa10b" candidate_uuid: "b091a31f41fa4314a6f422e74d7f222b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" is_pre_election: true
I20260929 03:10:39.521694 25944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786fce2d898a44d9b8fd4a563caaa10b" candidate_uuid: "b091a31f41fa4314a6f422e74d7f222b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871ab17fd294435ab216d83e61e4c7d" is_pre_election: true
I20260929 03:10:39.522038 25944 raft_consensus.cc:2454] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b091a31f41fa4314a6f422e74d7f222b in term 0.
I20260929 03:10:39.522038 26075 raft_consensus.cc:2454] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b091a31f41fa4314a6f422e74d7f222b in term 0.
I20260929 03:10:39.522651 26139 leader_election.cc:302] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8c94c62f370b4c0c9f62e1cdba50bd5e, b091a31f41fa4314a6f422e74d7f222b, d871ab17fd294435ab216d83e61e4c7d; no voters:
I20260929 03:10:39.522957 26275 raft_consensus.cc:2781] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:39.523048 26275 raft_consensus.cc:482] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:39.523111 26275 raft_consensus.cc:3037] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:39.524343 26275 raft_consensus.cc:504] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44979 } } peers { permanent_uuid: "b091a31f41fa4314a6f422e74d7f222b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45553 } } peers { permanent_uuid: "d871ab17fd294435ab216d83e61e4c7d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 37577 } }
I20260929 03:10:39.524734 26275 leader_election.cc:288] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [CANDIDATE]: Term 1 election: Requested vote from peers 8c94c62f370b4c0c9f62e1cdba50bd5e (127.20.174.66:44979), d871ab17fd294435ab216d83e61e4c7d (127.20.174.65:37577)
I20260929 03:10:39.525283 26075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786fce2d898a44d9b8fd4a563caaa10b" candidate_uuid: "b091a31f41fa4314a6f422e74d7f222b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e"
I20260929 03:10:39.525283 25944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786fce2d898a44d9b8fd4a563caaa10b" candidate_uuid: "b091a31f41fa4314a6f422e74d7f222b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871ab17fd294435ab216d83e61e4c7d"
I20260929 03:10:39.525434 26075 raft_consensus.cc:3037] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:39.525434 25944 raft_consensus.cc:3037] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:39.526885 25944 raft_consensus.cc:2454] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b091a31f41fa4314a6f422e74d7f222b in term 1.
I20260929 03:10:39.526885 26075 raft_consensus.cc:2454] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b091a31f41fa4314a6f422e74d7f222b in term 1.
I20260929 03:10:39.527267 26140 leader_election.cc:302] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b091a31f41fa4314a6f422e74d7f222b, d871ab17fd294435ab216d83e61e4c7d; no voters:
I20260929 03:10:39.527503 26275 raft_consensus.cc:2781] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:39.527757 26275 raft_consensus.cc:686] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 1 LEADER]: Becoming Leader. State: Replica: b091a31f41fa4314a6f422e74d7f222b, State: Running, Role: LEADER
I20260929 03:10:39.528136 26275 consensus_queue.cc:237] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44979 } } peers { permanent_uuid: "b091a31f41fa4314a6f422e74d7f222b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45553 } } peers { permanent_uuid: "d871ab17fd294435ab216d83e61e4c7d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 37577 } }
I20260929 03:10:39.531793 25801 catalog_manager.cc:5495] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b reported cstate change: term changed from 0 to 1, leader changed from <none> to b091a31f41fa4314a6f422e74d7f222b (127.20.174.67). New cstate: current_term: 1 leader_uuid: "b091a31f41fa4314a6f422e74d7f222b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b091a31f41fa4314a6f422e74d7f222b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871ab17fd294435ab216d83e61e4c7d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 37577 } health_report { overall_health: UNKNOWN } } }
W20260929 03:10:39.537447 26122 tablet.cc:2369] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:10:39.567857 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.566142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.20.174.67:45553)
0929 03:10:39.566301 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:39.566306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:39.566321 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:39.566571 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:39.566576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:39.566647 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:39.566745 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:39.567419 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:39.567423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:39.567425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:39.567439 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:39.567460 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:39.567477 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:39.567577 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:39.567646 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:10:39.567690 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:10:39.574662 25944 raft_consensus.cc:1261] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d [term 1 FOLLOWER]: Refusing update from remote peer b091a31f41fa4314a6f422e74d7f222b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:39.574662 26075 raft_consensus.cc:1261] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e [term 1 FOLLOWER]: Refusing update from remote peer b091a31f41fa4314a6f422e74d7f222b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:39.575512 26275 consensus_queue.cc:1035] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c94c62f370b4c0c9f62e1cdba50bd5e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:39.575740 26279 consensus_queue.cc:1035] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871ab17fd294435ab216d83e61e4c7d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 37577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:39.585110 26284 mvcc.cc:205] Tried to move back new op lower bound from 7334508296489324544 to 7334508296310284288. Current Snapshot: MvccSnapshot[applied={T|T < 7334508296489324544}]
I20260929 03:10:39.593977 25801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:39.594163 25801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:39.597295 25801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eba03eb77f794568b186ac5d803edbfb: Sending DeleteTablet for 3 replicas of tablet 786fce2d898a44d9b8fd4a563caaa10b
I20260929 03:10:39.598210 26055 tablet_service.cc:1514] Processing DeleteTablet for tablet 786fce2d898a44d9b8fd4a563caaa10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:39 UTC) from {username='slave'} at 127.0.0.1:35758
I20260929 03:10:39.598274 26186 tablet_service.cc:1514] Processing DeleteTablet for tablet 786fce2d898a44d9b8fd4a563caaa10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:39 UTC) from {username='slave'} at 127.0.0.1:60030
I20260929 03:10:39.598290 25924 tablet_service.cc:1514] Processing DeleteTablet for tablet 786fce2d898a44d9b8fd4a563caaa10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:39 UTC) from {username='slave'} at 127.0.0.1:35700
I20260929 03:10:39.599072 26294 tablet_replica.cc:330] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: stopping tablet replica
I20260929 03:10:39.599073 26293 tablet_replica.cc:330] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d: stopping tablet replica
I20260929 03:10:39.599475 26293 raft_consensus.cc:2229] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:39.599475 26294 raft_consensus.cc:2229] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:39.599663 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 25862
I20260929 03:10:39.599963 26293 raft_consensus.cc:2258] T 786fce2d898a44d9b8fd4a563caaa10b P d871ab17fd294435ab216d83e61e4c7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:39.599963 26294 raft_consensus.cc:2258] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:39.600440 26292 tablet_replica.cc:330] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: stopping tablet replica
I20260929 03:10:39.600792 26292 raft_consensus.cc:2229] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:39.601209 26292 raft_consensus.cc:2258] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:39.602730 26294 ts_tablet_manager.cc:1905] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:39.605888 26294 ts_tablet_manager.cc:1918] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:39.608767 26292 ts_tablet_manager.cc:1905] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:39.608969 26294 log.cc:1196] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/786fce2d898a44d9b8fd4a563caaa10b
I20260929 03:10:39.609416 26294 ts_tablet_manager.cc:1939] T 786fce2d898a44d9b8fd4a563caaa10b P 8c94c62f370b4c0c9f62e1cdba50bd5e: Deleting consensus metadata
I20260929 03:10:39.610906 25786 catalog_manager.cc:4841] TS 8c94c62f370b4c0c9f62e1cdba50bd5e (127.20.174.66:44979): tablet 786fce2d898a44d9b8fd4a563caaa10b (table test-table [id=3466c686573d4f8a80c076e4426bbf5e]) successfully deleted
I20260929 03:10:39.614065 26292 ts_tablet_manager.cc:1918] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:39.614282 26292 log.cc:1196] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/786fce2d898a44d9b8fd4a563caaa10b
I20260929 03:10:39.614821 26292 ts_tablet_manager.cc:1939] T 786fce2d898a44d9b8fd4a563caaa10b P b091a31f41fa4314a6f422e74d7f222b: Deleting consensus metadata
I20260929 03:10:39.615857 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 25993
W20260929 03:10:39.615860 25787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:37577 (error 108)
I20260929 03:10:39.617432 25787 catalog_manager.cc:4841] TS b091a31f41fa4314a6f422e74d7f222b (127.20.174.67:45553): tablet 786fce2d898a44d9b8fd4a563caaa10b (table test-table [id=3466c686573d4f8a80c076e4426bbf5e]) successfully deleted
I20260929 03:10:39.622916 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 26124
I20260929 03:10:39.623673 26266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.623163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:37577 (local address 127.0.0.1:35704)
0929 03:10:39.623466 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:39.623557 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:37577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
W20260929 03:10:39.623934 25787 catalog_manager.cc:4584] TS d871ab17fd294435ab216d83e61e4c7d (127.20.174.65:37577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 786fce2d898a44d9b8fd4a563caaa10b: Network error: Client connection negotiation failed: client connection to 127.20.174.65:37577: connect: Connection refused (error 111)
I20260929 03:10:39.630144 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 25771
2026-09-29T03:10:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1226 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 03:10:39.653383 21177 mini_oidc.cc:152] Starting JWKS server
I20260929 03:10:39.655331 21177 webserver.cc:460] Webserver started at https://127.0.0.1:34521/ using document root <none> and password file <none>
I20260929 03:10:39.655499 21177 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:10:39.655757 21177 webserver.cc:460] Webserver started at http://127.0.0.1:37255/ using document root <none> and password file <none>
2026-09-29T03:10:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:39Z Disabled control of system clock
I20260929 03:10:39.665797 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44549
--webserver_interface=127.20.174.126
--webserver_port=0
--builtin_ntp_servers=127.20.174.84:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:44549
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34521/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/testchainca.cert with env {}
W20260929 03:10:39.796197 26301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:39.796528 26301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:39.796576 26301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:39.801368 26301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:39.801491 26301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:39.801524 26301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:39.801560 26301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:39.801595 26301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:10:39.801635 26301 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:10:39.801651 26301 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34521/jwks.json
I20260929 03:10:39.807962 26301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:44549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34521/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:39.809099 26301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:39.810171 26301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:39.819690 26307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:39.819787 26306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:39.819684 26309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:39.819801 26301 server_base.cc:1034] running on GCE node
I20260929 03:10:39.820633 26301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:39.821874 26301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:39.823066 26301 hybrid_clock.cc:648] HybridClock initialized: now 1790651439823036 us; error 59 us; skew 500 ppm
I20260929 03:10:39.825572 26301 webserver.cc:460] Webserver started at http://127.20.174.126:41127/ using document root <none> and password file <none>
I20260929 03:10:39.826246 26301 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:39.826352 26301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:39.826633 26301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:39.828866 26301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "036457b899aa4da88885ce3715648fb0"
format_stamp: "Formatted at 2026-09-29 03:10:39 on dist-test-slave-k1dg"
server_key: "13767dd9bc9c2dc6ca69fd3868b93023"
server_key_iv: "3c617903bc54ad9ba1d54243bfdf46b3"
server_key_version: "encryptionkey@0"
I20260929 03:10:39.829491 26301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "036457b899aa4da88885ce3715648fb0"
format_stamp: "Formatted at 2026-09-29 03:10:39 on dist-test-slave-k1dg"
server_key: "13767dd9bc9c2dc6ca69fd3868b93023"
server_key_iv: "3c617903bc54ad9ba1d54243bfdf46b3"
server_key_version: "encryptionkey@0"
I20260929 03:10:39.833580 26301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 03:10:39.836416 26315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:39.838027 26301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:39.838195 26301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "036457b899aa4da88885ce3715648fb0"
format_stamp: "Formatted at 2026-09-29 03:10:39 on dist-test-slave-k1dg"
server_key: "13767dd9bc9c2dc6ca69fd3868b93023"
server_key_iv: "3c617903bc54ad9ba1d54243bfdf46b3"
server_key_version: "encryptionkey@0"
I20260929 03:10:39.838332 26301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:39.864432 26301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:39.865235 26301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:39.865458 26301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:39.873096 26301 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:44549
I20260929 03:10:39.873109 26367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:44549 every 8 connection(s)
I20260929 03:10:39.874178 26301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:39.877666 26368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:39.883098 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 26301
I20260929 03:10:39.883278 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:39.883560 21177 external_mini_cluster.cc:1428] Setting key 395c57f396b607ece043d71242931a09
I20260929 03:10:39.883730 26368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0: Bootstrap starting.
I20260929 03:10:39.886509 26368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:39.887398 26368 log.cc:824] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:39.889554 26368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0: No bootstrap required, opened a new log
I20260929 03:10:39.893429 26368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036457b899aa4da88885ce3715648fb0" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44549 } }
I20260929 03:10:39.893642 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.884883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.20.174.126:44549)
0929 03:10:39.885652 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:39.885668 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:39.885707 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:10:39.886541 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:39.886562 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:39.887097 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:39.887479 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:39.887957 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:39.888954 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:39.890012 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:39.890584 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:39.890795 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:39.891699 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:39.891710 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:39.891724 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:39.892147 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:39.892205 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:39.892226 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:39.892343 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:39.892467 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 03:10:39.892817 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":141,"threads_started":1}
I20260929 03:10:39.893728 26368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:39.893795 26368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036457b899aa4da88885ce3715648fb0, State: Initialized, Role: FOLLOWER
I20260929 03:10:39.894315 26368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036457b899aa4da88885ce3715648fb0" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44549 } }
I20260929 03:10:39.894526 26368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:39.894593 26368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:39.894690 26368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:39.895713 26368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036457b899aa4da88885ce3715648fb0" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44549 } }
I20260929 03:10:39.896106 26368 leader_election.cc:302] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 036457b899aa4da88885ce3715648fb0; no voters:
I20260929 03:10:39.896505 26368 leader_election.cc:288] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:39.896639 26373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:39.897001 26373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [term 1 LEADER]: Becoming Leader. State: Replica: 036457b899aa4da88885ce3715648fb0, State: Running, Role: LEADER
I20260929 03:10:39.897094 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:10:39.897344 26373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036457b899aa4da88885ce3715648fb0" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44549 } }
I20260929 03:10:39.897960 26368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:39.898742 26375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 036457b899aa4da88885ce3715648fb0. Latest consensus state: current_term: 1 leader_uuid: "036457b899aa4da88885ce3715648fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036457b899aa4da88885ce3715648fb0" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44549 } } }
I20260929 03:10:39.898877 26375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:39.899477 26374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "036457b899aa4da88885ce3715648fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036457b899aa4da88885ce3715648fb0" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44549 } } }
I20260929 03:10:39.899576 26374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0 [sys.catalog]: This master's current role is: LEADER
W20260929 03:10:39.903121 26388 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:10:39.903219 26388 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:10:39.903317 26380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:39.904030 26380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:39.909584 26380 catalog_manager.cc:1347] Generated new cluster ID: 4b50c8d0804a49039381fdd82b47bd41
I20260929 03:10:39.909660 26380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:39.929140 26380 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:39.930145 26380 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:39.935127 26380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 036457b899aa4da88885ce3715648fb0: Generated new TSK 0
I20260929 03:10:39.935880 26380 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:10:39.966009 26398 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 03:10:39.966159 26398 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:10:39.966809 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:39.948333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.20.174.126:44549)
0929 03:10:39.948486 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:39.948490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:39.948699 (+ 209us) server_negotiation.cc:407] Connection header received
0929 03:10:39.948918 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:39.948922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:39.948983 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:39.949106 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:10:39.949394 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:39.950267 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:39.951321 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:39.951647 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:39.951759 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:39.952167 (+ 408us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:10:39.966226 (+ 14059us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:10:39.966644 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 03:10:39.966918 26371 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 03:10:39.968185 26397 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:39.948246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44549 (local address 127.0.0.1:55344)
0929 03:10:39.948592 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:39.948606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:39.948733 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:39.949133 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:39.949136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:39.949143 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:10:39.949272 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:39.949277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:39.950418 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:39.950496 (+ 78us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:39.951181 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:39.951192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:39.951787 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:39.951790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:39.951870 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:39.951948 (+ 78us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:10:39.966631 (+ 14683us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:10:39.966872 (+ 241us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.174.126:44549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.174.126:44549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 03:10:39.969197 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 26301
2026-09-29T03:10:39Z chronyd exiting
I20260929 03:10:40.006685 21177 test_util.cc:175] -----------------------------------------------
I20260929 03:10:40.006798 21177 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651413504048-21177-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (355 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 03:10:40.008070 21177 mini_oidc.cc:152] Starting JWKS server
I20260929 03:10:40.009038 21177 webserver.cc:460] Webserver started at https://127.0.0.1:36567/ using document root <none> and password file <none>
I20260929 03:10:40.009208 21177 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:10:40.009567 21177 webserver.cc:460] Webserver started at http://127.0.0.1:33021/ using document root <none> and password file <none>
2026-09-29T03:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:40Z Disabled control of system clock
I20260929 03:10:40.019791 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:39109
--webserver_interface=127.20.174.126
--webserver_port=0
--builtin_ntp_servers=127.20.174.84:38223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:39109
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36567/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/testchainca.cert with env {}
W20260929 03:10:40.136018 26404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:40.136351 26404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:40.136401 26404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:40.140776 26404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:40.140862 26404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:40.140878 26404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:40.140897 26404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:40.140914 26404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:10:40.140931 26404 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:10:40.140945 26404 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36567/jwks.json
I20260929 03:10:40.147007 26404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:39109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:39109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36567/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:40.148238 26404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:40.149327 26404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:40.158715 26409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:40.158830 26404 server_base.cc:1034] running on GCE node
W20260929 03:10:40.158711 26410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:40.158741 26412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:40.159627 26404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:40.160933 26404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:40.162137 26404 hybrid_clock.cc:648] HybridClock initialized: now 1790651440162114 us; error 40 us; skew 500 ppm
I20260929 03:10:40.164304 26404 webserver.cc:460] Webserver started at http://127.20.174.126:35001/ using document root <none> and password file <none>
I20260929 03:10:40.164932 26404 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:40.165032 26404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:40.165267 26404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:40.167511 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "a4b2488093c04fdaac0f8ad07d77a1c4"
format_stamp: "Formatted at 2026-09-29 03:10:40 on dist-test-slave-k1dg"
server_key: "d588c705bc269a47f10c6bfd9480844d"
server_key_iv: "75059db484ce034a0ddb13a9e19ad2f5"
server_key_version: "encryptionkey@0"
I20260929 03:10:40.168138 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "a4b2488093c04fdaac0f8ad07d77a1c4"
format_stamp: "Formatted at 2026-09-29 03:10:40 on dist-test-slave-k1dg"
server_key: "d588c705bc269a47f10c6bfd9480844d"
server_key_iv: "75059db484ce034a0ddb13a9e19ad2f5"
server_key_version: "encryptionkey@0"
I20260929 03:10:40.172158 26404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:40.174979 26418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:40.176491 26404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:40.176661 26404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "a4b2488093c04fdaac0f8ad07d77a1c4"
format_stamp: "Formatted at 2026-09-29 03:10:40 on dist-test-slave-k1dg"
server_key: "d588c705bc269a47f10c6bfd9480844d"
server_key_iv: "75059db484ce034a0ddb13a9e19ad2f5"
server_key_version: "encryptionkey@0"
I20260929 03:10:40.176803 26404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:40.192066 26404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:40.192919 26404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:40.193150 26404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:40.201587 26404 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:39109
I20260929 03:10:40.201608 26470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:39109 every 8 connection(s)
I20260929 03:10:40.202773 26404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:40.206907 26471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:40.207190 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 26404
I20260929 03:10:40.207367 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:40.207748 21177 external_mini_cluster.cc:1428] Setting key ffa2ed2f960cb06ddb2641d7beaaae67
I20260929 03:10:40.214135 26471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4: Bootstrap starting.
I20260929 03:10:40.216773 26471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:40.217814 26471 log.cc:824] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:40.218153 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:40.209386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.20.174.126:39109)
0929 03:10:40.210357 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:40.210373 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:40.210412 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:10:40.211184 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:40.211203 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:40.211668 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:40.212061 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:40.212519 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.213539 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.214627 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.215149 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.215344 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:40.216265 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:40.216270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:40.216284 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:40.216702 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:40.216763 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:40.216785 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:40.216906 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:40.217036 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 03:10:40.217326 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":178,"threads_started":1}
I20260929 03:10:40.220453 26471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4: No bootstrap required, opened a new log
I20260929 03:10:40.221235 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:10:40.224661 26471 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b2488093c04fdaac0f8ad07d77a1c4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39109 } }
I20260929 03:10:40.224933 26471 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:40.225016 26471 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b2488093c04fdaac0f8ad07d77a1c4, State: Initialized, Role: FOLLOWER
I20260929 03:10:40.225587 26471 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b2488093c04fdaac0f8ad07d77a1c4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39109 } }
I20260929 03:10:40.225747 26471 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:40.225817 26471 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:40.225940 26471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:40.227283 26471 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b2488093c04fdaac0f8ad07d77a1c4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39109 } }
I20260929 03:10:40.227682 26471 leader_election.cc:302] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4b2488093c04fdaac0f8ad07d77a1c4; no voters:
I20260929 03:10:40.228057 26471 leader_election.cc:288] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:40.228323 26476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:40.228579 26476 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [term 1 LEADER]: Becoming Leader. State: Replica: a4b2488093c04fdaac0f8ad07d77a1c4, State: Running, Role: LEADER
I20260929 03:10:40.228921 26476 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b2488093c04fdaac0f8ad07d77a1c4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39109 } }
I20260929 03:10:40.229460 26471 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:40.230754 26478 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b2488093c04fdaac0f8ad07d77a1c4. Latest consensus state: current_term: 1 leader_uuid: "a4b2488093c04fdaac0f8ad07d77a1c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b2488093c04fdaac0f8ad07d77a1c4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39109 } } }
I20260929 03:10:40.230813 26477 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4b2488093c04fdaac0f8ad07d77a1c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b2488093c04fdaac0f8ad07d77a1c4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39109 } } }
I20260929 03:10:40.230913 26478 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:40.230916 26477 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:40.231360 26485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:40.234930 26485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:40.241843 26485 catalog_manager.cc:1347] Generated new cluster ID: 53d013e6ce474d1298d579b33e4771ec
I20260929 03:10:40.241943 26485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:40.264037 26485 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:40.265414 26485 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:40.276118 26485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4b2488093c04fdaac0f8ad07d77a1c4: Generated new TSK 0
I20260929 03:10:40.277102 26485 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:10:40.301729 26501 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 03:10:40.301862 26501 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:10:40.302663 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:40.285211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.20.174.126:39109)
0929 03:10:40.285391 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:40.285396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:40.285603 (+ 207us) server_negotiation.cc:407] Connection header received
0929 03:10:40.285785 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:40.285789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:40.285845 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:40.285955 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:10:40.286248 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.286951 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.288227 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.288460 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.288548 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:40.288946 (+ 398us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:10:40.302179 (+ 13233us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36567/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:10:40.302521 (+ 342us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40042: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36567/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 03:10:40.302752 26474 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40042: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36567/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 03:10:40.302966 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:40.285126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39109 (local address 127.0.0.1:40042)
0929 03:10:40.285499 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:40.285516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:40.285644 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:40.285990 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:40.285994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:40.286002 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:10:40.286121 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:40.286127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.287084 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.287089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:40.288081 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:40.288094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.288595 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.288599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:40.288699 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:40.288731 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:10:40.302411 (+ 13680us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36567/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:10:40.302661 (+ 250us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.174.126:39109: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36567/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":159,"threads_started":1}
I20260929 03:10:40.303864 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 26404
2026-09-29T03:10:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (335 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 03:10:40.343930 21177 mini_oidc.cc:152] Starting JWKS server
I20260929 03:10:40.344880 21177 webserver.cc:460] Webserver started at https://127.0.0.1:33865/ using document root <none> and password file <none>
I20260929 03:10:40.345046 21177 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:10:40.345337 21177 webserver.cc:460] Webserver started at http://127.0.0.1:34805/ using document root <none> and password file <none>
2026-09-29T03:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:40Z Disabled control of system clock
I20260929 03:10:40.355613 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40041
--webserver_interface=127.20.174.126
--webserver_port=0
--builtin_ntp_servers=127.20.174.84:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:40041
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33865/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:10:40.477279 26507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:40.477614 26507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:40.477659 26507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:40.482210 26507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:40.482290 26507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:40.482306 26507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:40.482324 26507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:40.482342 26507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:10:40.482358 26507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:10:40.482372 26507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33865/jwks.json
I20260929 03:10:40.487594 26507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:40041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33865/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:40.488801 26507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:40.490227 26507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:40.499507 26513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:40.499496 26512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:40.499787 26507 server_base.cc:1034] running on GCE node
W20260929 03:10:40.499496 26515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:40.500504 26507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:40.501767 26507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:40.503072 26507 hybrid_clock.cc:648] HybridClock initialized: now 1790651440502964 us; error 122 us; skew 500 ppm
I20260929 03:10:40.505326 26507 webserver.cc:460] Webserver started at http://127.20.174.126:39833/ using document root <none> and password file <none>
I20260929 03:10:40.505976 26507 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:40.506081 26507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:40.506343 26507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:40.508602 26507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "17c896e25e3f460191de94d9dda66232"
format_stamp: "Formatted at 2026-09-29 03:10:40 on dist-test-slave-k1dg"
server_key: "496518dd64dea987d9ed2e5a80c99ef7"
server_key_iv: "fabfea9ac236b2852fd658e3ada14d9b"
server_key_version: "encryptionkey@0"
I20260929 03:10:40.509233 26507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "17c896e25e3f460191de94d9dda66232"
format_stamp: "Formatted at 2026-09-29 03:10:40 on dist-test-slave-k1dg"
server_key: "496518dd64dea987d9ed2e5a80c99ef7"
server_key_iv: "fabfea9ac236b2852fd658e3ada14d9b"
server_key_version: "encryptionkey@0"
I20260929 03:10:40.513492 26507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:10:40.516824 26521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:40.518395 26507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:10:40.518646 26507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "17c896e25e3f460191de94d9dda66232"
format_stamp: "Formatted at 2026-09-29 03:10:40 on dist-test-slave-k1dg"
server_key: "496518dd64dea987d9ed2e5a80c99ef7"
server_key_iv: "fabfea9ac236b2852fd658e3ada14d9b"
server_key_version: "encryptionkey@0"
I20260929 03:10:40.518875 26507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:40.536361 26507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:40.537168 26507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:40.537396 26507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:40.545624 26507 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:40041
I20260929 03:10:40.545625 26573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:40041 every 8 connection(s)
I20260929 03:10:40.546741 26507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:40.550227 26574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:40.552835 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 26507
I20260929 03:10:40.552953 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:40.553258 21177 external_mini_cluster.cc:1428] Setting key 634f32f74ef483adf3c70470aae3b4dd
I20260929 03:10:40.557720 26574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232: Bootstrap starting.
I20260929 03:10:40.560582 26574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:40.561615 26574 log.cc:824] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:40.563959 26574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232: No bootstrap required, opened a new log
I20260929 03:10:40.564720 26577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:40.554888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.20.174.126:40041)
0929 03:10:40.555704 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:40.555746 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:40.555796 (+ 50us) server_negotiation.cc:407] Connection header received
0929 03:10:40.556664 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:40.556690 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:40.557251 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:40.557656 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:40.558157 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.559656 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.560680 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.561416 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.561612 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:40.562698 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:40.562705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:40.562720 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:40.563204 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:40.563266 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:40.563289 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:40.563422 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:40.563598 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 03:10:40.563898 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":128,"threads_started":1}
I20260929 03:10:40.568305 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:10:40.568655 26574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c896e25e3f460191de94d9dda66232" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40041 } }
I20260929 03:10:40.568917 26574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:40.569006 26574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c896e25e3f460191de94d9dda66232, State: Initialized, Role: FOLLOWER
I20260929 03:10:40.569512 26574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c896e25e3f460191de94d9dda66232" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40041 } }
I20260929 03:10:40.569690 26574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:40.569758 26574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:40.569860 26574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:40.571038 26574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c896e25e3f460191de94d9dda66232" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40041 } }
I20260929 03:10:40.571400 26574 leader_election.cc:302] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17c896e25e3f460191de94d9dda66232; no voters:
I20260929 03:10:40.571730 26574 leader_election.cc:288] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:40.571905 26579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:40.572144 26579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [term 1 LEADER]: Becoming Leader. State: Replica: 17c896e25e3f460191de94d9dda66232, State: Running, Role: LEADER
I20260929 03:10:40.572525 26579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c896e25e3f460191de94d9dda66232" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40041 } }
I20260929 03:10:40.573103 26574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:40.574323 26580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17c896e25e3f460191de94d9dda66232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c896e25e3f460191de94d9dda66232" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40041 } } }
I20260929 03:10:40.574482 26580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:40.574919 26588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:40.574878 26581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17c896e25e3f460191de94d9dda66232. Latest consensus state: current_term: 1 leader_uuid: "17c896e25e3f460191de94d9dda66232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c896e25e3f460191de94d9dda66232" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40041 } } }
I20260929 03:10:40.575033 26581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:40.577904 26588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:40.584170 26588 catalog_manager.cc:1347] Generated new cluster ID: 167f164b9c7e4396930e221bdde87af1
I20260929 03:10:40.584265 26588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:40.590972 26588 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:40.592269 26588 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:40.604462 26588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17c896e25e3f460191de94d9dda66232: Generated new TSK 0
I20260929 03:10:40.605530 26588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:10:40.627267 26577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:40.621200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.20.174.126:40041)
0929 03:10:40.621351 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:40.621356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:40.621481 (+ 125us) server_negotiation.cc:407] Connection header received
0929 03:10:40.621647 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:40.621651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:40.621715 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:40.621863 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:10:40.622157 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.623085 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.624414 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.624784 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.624897 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:40.625219 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:10:40.626874 (+ 1655us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:10:40.627120 (+ 246us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47670: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 03:10:40.627345 26577 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47670: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 03:10:40.629524 26603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:40.621092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40041 (local address 127.0.0.1:47670)
0929 03:10:40.621374 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:40.621397 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:40.621503 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:40.621916 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:40.621920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:40.621932 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:10:40.622042 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:40.622048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.623248 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.623252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:40.624280 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:40.624291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.624929 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.624932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:40.625004 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:40.625030 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:10:40.627047 (+ 2017us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:10:40.629338 (+ 2291us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.174.126:40041: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":2042,"thread_start_us":108,"threads_started":1}
I20260929 03:10:40.630215 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 26507
E20260929 03:10:40.638953 26604 webserver.cc:533] Webserver: sslize failed:
E20260929 03:10:40.639037 26604 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T03:10:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (325 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T03:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:40Z Disabled control of system clock
I20260929 03:10:40.679697 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44157
--webserver_interface=127.20.174.126
--webserver_port=0
--builtin_ntp_servers=127.20.174.84:41559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:44157
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/testchainca.cert with env {}
W20260929 03:10:40.798844 26608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:40.799177 26608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:40.799221 26608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:40.804267 26608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:40.804353 26608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:40.804373 26608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:40.804390 26608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:40.804406 26608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:10:40.804469 26608 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:10:40.804488 26608 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 03:10:40.810968 26608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:44157
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:40.812206 26608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:40.813341 26608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:40.822818 26613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:40.822865 26614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:40.823000 26608 server_base.cc:1034] running on GCE node
W20260929 03:10:40.822818 26616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:40.823805 26608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:40.825037 26608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:40.826256 26608 hybrid_clock.cc:648] HybridClock initialized: now 1790651440826219 us; error 56 us; skew 500 ppm
I20260929 03:10:40.828299 26608 webserver.cc:460] Webserver started at http://127.20.174.126:38081/ using document root <none> and password file <none>
I20260929 03:10:40.828938 26608 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:40.829036 26608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:40.829250 26608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:40.831584 26608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "5a5993c29f414aeba7aba047e92f5b94"
format_stamp: "Formatted at 2026-09-29 03:10:40 on dist-test-slave-k1dg"
server_key: "2432cfad5a18e3d73c968c583a03ffc4"
server_key_iv: "19cfd206c0ea3212f5f1a3294899f94d"
server_key_version: "encryptionkey@0"
I20260929 03:10:40.832216 26608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "5a5993c29f414aeba7aba047e92f5b94"
format_stamp: "Formatted at 2026-09-29 03:10:40 on dist-test-slave-k1dg"
server_key: "2432cfad5a18e3d73c968c583a03ffc4"
server_key_iv: "19cfd206c0ea3212f5f1a3294899f94d"
server_key_version: "encryptionkey@0"
I20260929 03:10:40.836434 26608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:40.839221 26622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:40.840931 26608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:40.841128 26608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "5a5993c29f414aeba7aba047e92f5b94"
format_stamp: "Formatted at 2026-09-29 03:10:40 on dist-test-slave-k1dg"
server_key: "2432cfad5a18e3d73c968c583a03ffc4"
server_key_iv: "19cfd206c0ea3212f5f1a3294899f94d"
server_key_version: "encryptionkey@0"
I20260929 03:10:40.841285 26608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:40.867159 26608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:40.868044 26608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:40.868281 26608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:40.876253 26608 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:44157
I20260929 03:10:40.876274 26674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:44157 every 8 connection(s)
I20260929 03:10:40.877393 26608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:40.880797 26675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:40.886725 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 26608
I20260929 03:10:40.886878 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:40.887171 21177 external_mini_cluster.cc:1428] Setting key 0e18e5877032c9fd16bca6721029d5ee
I20260929 03:10:40.887310 26675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94: Bootstrap starting.
I20260929 03:10:40.890184 26675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:40.891077 26675 log.cc:824] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:40.893177 26675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94: No bootstrap required, opened a new log
I20260929 03:10:40.897444 26675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5993c29f414aeba7aba047e92f5b94" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44157 } }
I20260929 03:10:40.897671 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:40.888733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.20.174.126:44157)
0929 03:10:40.889516 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:40.889531 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:40.889570 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:10:40.890363 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:40.890387 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:40.890978 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:40.891438 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:40.891926 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.892937 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.894010 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.894593 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.894773 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:40.895704 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:40.895709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:40.895724 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:10:40.896228 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:40.896289 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:10:40.896313 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:40.896425 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:40.896583 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:10:40.896867 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":139,"threads_started":1}
I20260929 03:10:40.897729 26675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:40.897811 26675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5993c29f414aeba7aba047e92f5b94, State: Initialized, Role: FOLLOWER
I20260929 03:10:40.898341 26675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5993c29f414aeba7aba047e92f5b94" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44157 } }
I20260929 03:10:40.898518 26675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:40.898574 26675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:40.898670 26675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:40.899705 26675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5993c29f414aeba7aba047e92f5b94" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44157 } }
I20260929 03:10:40.900074 26675 leader_election.cc:302] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a5993c29f414aeba7aba047e92f5b94; no voters:
I20260929 03:10:40.900446 26675 leader_election.cc:288] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:40.900573 26680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:40.900797 26680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [term 1 LEADER]: Becoming Leader. State: Replica: 5a5993c29f414aeba7aba047e92f5b94, State: Running, Role: LEADER
I20260929 03:10:40.901134 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:10:40.901219 26680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5993c29f414aeba7aba047e92f5b94" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44157 } }
I20260929 03:10:40.901914 26675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:40.902961 26681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a5993c29f414aeba7aba047e92f5b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5993c29f414aeba7aba047e92f5b94" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44157 } } }
I20260929 03:10:40.903115 26681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:40.903432 26682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5993c29f414aeba7aba047e92f5b94. Latest consensus state: current_term: 1 leader_uuid: "5a5993c29f414aeba7aba047e92f5b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5993c29f414aeba7aba047e92f5b94" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44157 } } }
I20260929 03:10:40.903527 26682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:40.903563 26689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:40.907044 26689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:40.913219 26689 catalog_manager.cc:1347] Generated new cluster ID: c0df597e78484a0fb45a64ad09b44885
I20260929 03:10:40.913339 26689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:40.929419 26689 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:40.930639 26689 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:40.941919 26689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a5993c29f414aeba7aba047e92f5b94: Generated new TSK 0
I20260929 03:10:40.942780 26689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:10:41.035220 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:40.955009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.20.174.126:44157)
0929 03:10:40.955162 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:40.955167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:40.955300 (+ 133us) server_negotiation.cc:407] Connection header received
0929 03:10:40.955530 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:40.955534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:40.955592 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:40.955702 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:10:40.956041 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.956801 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.958275 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.958584 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.958679 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:40.959110 (+ 431us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:10:41.034793 (+ 75683us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:10:41.035099 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57546: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 03:10:41.035351 26678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57546: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 03:10:41.035537 26704 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:40.954803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44157 (local address 127.0.0.1:57546)
0929 03:10:40.955164 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:40.955181 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:40.955366 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:40.955733 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:40.955737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:40.955746 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:10:40.955880 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:40.955887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.956945 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.956949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:40.958129 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:40.958140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:40.958719 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:40.958724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:40.958825 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:40.958857 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:10:41.035078 (+ 76221us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:10:41.035318 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.174.126:44157: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":143,"threads_started":1}
I20260929 03:10:41.036213 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 26608
2026-09-29T03:10:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (386 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (135 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (136 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:41 dist-test-slave-k1dg krb5kdc[26716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:41 dist-test-slave-k1dg krb5kdc[26716](info): set up 2 sockets
Sep 29 03:10:41 dist-test-slave-k1dg krb5kdc[26716](info): commencing operation
krb5kdc: starting...
W20260929 03:10:42.403340 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:42 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:42.537756 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40787
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:40787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:42.654497 26729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:42.654992 26729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:42.655083 26729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:42.659396 26729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:42.659516 26729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:42.659554 26729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:42.659595 26729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:42.659631 26729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:42.665580 26729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:40787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:42.666908 26729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:42.668054 26729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:42.677387 26737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:42.677510 26729 server_base.cc:1034] running on GCE node
W20260929 03:10:42.677531 26734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:42.677395 26735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:42.678350 26729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:42.679616 26729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:42.680920 26729 hybrid_clock.cc:648] HybridClock initialized: now 1790651442680906 us; error 31 us; skew 500 ppm
Sep 29 03:10:42 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651442, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:42.691841 26729 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:42.692929 26729 webserver.cc:460] Webserver started at http://127.20.174.126:39827/ using document root <none> and password file <none>
I20260929 03:10:42.693563 26729 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:42.693689 26729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:42.693943 26729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:42.696236 26729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "ed7221d7cf2b42e6b379c843897c3c9e"
format_stamp: "Formatted at 2026-09-29 03:10:42 on dist-test-slave-k1dg"
server_key: "d8702c0c087ef949bf17e4938726b1c5"
server_key_iv: "6c0972c70bdc4de91f7370c0e5f930cd"
server_key_version: "encryptionkey@0"
I20260929 03:10:42.696853 26729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "ed7221d7cf2b42e6b379c843897c3c9e"
format_stamp: "Formatted at 2026-09-29 03:10:42 on dist-test-slave-k1dg"
server_key: "d8702c0c087ef949bf17e4938726b1c5"
server_key_iv: "6c0972c70bdc4de91f7370c0e5f930cd"
server_key_version: "encryptionkey@0"
I20260929 03:10:42.701184 26729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 03:10:42.703991 26744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:42.705447 26729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:42.705613 26729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "ed7221d7cf2b42e6b379c843897c3c9e"
format_stamp: "Formatted at 2026-09-29 03:10:42 on dist-test-slave-k1dg"
server_key: "d8702c0c087ef949bf17e4938726b1c5"
server_key_iv: "6c0972c70bdc4de91f7370c0e5f930cd"
server_key_version: "encryptionkey@0"
I20260929 03:10:42.705742 26729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:42.717818 26729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:42.721150 26729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:42.721392 26729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:42.729632 26729 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:40787
I20260929 03:10:42.729661 26796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:40787 every 8 connection(s)
I20260929 03:10:42.730788 26729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:42.733781 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:42.735067 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 26729
I20260929 03:10:42.735208 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:42.735526 21177 external_mini_cluster.cc:1428] Setting key f25a06262254d363953dceb9ad0c9bef
I20260929 03:10:42.740645 26797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e: Bootstrap starting.
I20260929 03:10:42.743149 26797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:42.744145 26797 log.cc:824] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:10:42 dist-test-slave-k1dg krb5kdc[26716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:42.746714 26797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e: No bootstrap required, opened a new log
I20260929 03:10:42.751902 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:42.737226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.20.174.126:40787)
0929 03:10:42.737724 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:42.737741 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:42.737780 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:10:42.738650 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:42.738673 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:42.739075 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:42.739489 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:42.740392 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:42.741462 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:42.742488 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:42.742994 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:42.743184 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:42.746054 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:42.746061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:42.746078 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:42.746111 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:42.748233 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:42.748722 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:42.748731 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:42.748738 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:42.748891 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:42.749185 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:42.749189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:42.749192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:42.749882 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:42.750081 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:42.750366 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 03:10:42.750798 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":140,"threads_started":1}
I20260929 03:10:42.751843 26797 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7221d7cf2b42e6b379c843897c3c9e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40787 } }
I20260929 03:10:42.752183 26797 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:42.752256 26797 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7221d7cf2b42e6b379c843897c3c9e, State: Initialized, Role: FOLLOWER
I20260929 03:10:42.752774 26797 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7221d7cf2b42e6b379c843897c3c9e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40787 } }
I20260929 03:10:42.752971 26797 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:42.753043 26797 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:42.753149 26797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:42.754212 26797 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7221d7cf2b42e6b379c843897c3c9e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40787 } }
I20260929 03:10:42.754603 26797 leader_election.cc:302] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed7221d7cf2b42e6b379c843897c3c9e; no voters:
I20260929 03:10:42.754892 26797 leader_election.cc:288] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:42.755038 26802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:42.755268 26802 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [term 1 LEADER]: Becoming Leader. State: Replica: ed7221d7cf2b42e6b379c843897c3c9e, State: Running, Role: LEADER
I20260929 03:10:42.755640 26802 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7221d7cf2b42e6b379c843897c3c9e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40787 } }
I20260929 03:10:42.756194 26797 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:42.757360 26804 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed7221d7cf2b42e6b379c843897c3c9e. Latest consensus state: current_term: 1 leader_uuid: "ed7221d7cf2b42e6b379c843897c3c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7221d7cf2b42e6b379c843897c3c9e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40787 } } }
I20260929 03:10:42.757519 26804 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:42.757783 26803 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed7221d7cf2b42e6b379c843897c3c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7221d7cf2b42e6b379c843897c3c9e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40787 } } }
I20260929 03:10:42.757889 26811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:42.757929 26803 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:42.761241 26811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:42.768002 26811 catalog_manager.cc:1347] Generated new cluster ID: 848f09b1b85c49198604a9b691c864a6
I20260929 03:10:42.768124 26811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:42.784447 26811 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:42.785585 26811 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:42.792307 26811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed7221d7cf2b42e6b379c843897c3c9e: Generated new TSK 0
I20260929 03:10:42.793118 26811 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:10:42.835047 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40787
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:42.967918 26823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:42.968210 26823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:42.968257 26823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:42.972545 26823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:42.972611 26823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:42.972673 26823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:42.978154 26823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:42.979389 26823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:42.980401 26823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:42.990664 26829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:42.990664 26828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:42.990726 26831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:42.990993 26823 server_base.cc:1034] running on GCE node
I20260929 03:10:42.991503 26823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:42.992229 26823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:42.993443 26823 hybrid_clock.cc:648] HybridClock initialized: now 1790651442993403 us; error 54 us; skew 500 ppm
Sep 29 03:10:42 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651442, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:42.996510 26823 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:10:42.997465 26823 webserver.cc:460] Webserver started at http://127.20.174.65:35681/ using document root <none> and password file <none>
I20260929 03:10:42.998106 26823 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:42.998198 26823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:42.998471 26823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:43.000689 26823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "c318604d679d42f3982f22660895aabd"
format_stamp: "Formatted at 2026-09-29 03:10:42 on dist-test-slave-k1dg"
server_key: "9cde290b6da92293ee4c6f17e1902a42"
server_key_iv: "fb43b8d49dc89272df9e02211e2ff08e"
server_key_version: "encryptionkey@0"
I20260929 03:10:43.001286 26823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "c318604d679d42f3982f22660895aabd"
format_stamp: "Formatted at 2026-09-29 03:10:42 on dist-test-slave-k1dg"
server_key: "9cde290b6da92293ee4c6f17e1902a42"
server_key_iv: "fb43b8d49dc89272df9e02211e2ff08e"
server_key_version: "encryptionkey@0"
I20260929 03:10:43.005347 26823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:10:43.007965 26838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:43.009600 26823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:43.009858 26823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "c318604d679d42f3982f22660895aabd"
format_stamp: "Formatted at 2026-09-29 03:10:42 on dist-test-slave-k1dg"
server_key: "9cde290b6da92293ee4c6f17e1902a42"
server_key_iv: "fb43b8d49dc89272df9e02211e2ff08e"
server_key_version: "encryptionkey@0"
I20260929 03:10:43.010001 26823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:43.038184 26823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:43.042214 26823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:43.042470 26823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:43.043195 26823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:43.044121 26823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:43.044196 26823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:43.044268 26823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:43.044319 26823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:43.054299 26823 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:35335
I20260929 03:10:43.054323 26951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:35335 every 8 connection(s)
I20260929 03:10:43.055307 26823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:43.063370 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 26823
I20260929 03:10:43.063484 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:43.063805 21177 external_mini_cluster.cc:1428] Setting key b6f40321478308b9c466453dcbba0068
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651442, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:43.071475 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:43.057410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:53851 (local address 127.20.174.126:40787)
0929 03:10:43.057583 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:43.057589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:43.058788 (+ 1199us) server_negotiation.cc:407] Connection header received
0929 03:10:43.060318 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:43.060321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:43.060385 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:43.060495 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:43.061797 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.062341 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.063630 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.063874 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.063998 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:43.066932 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:43.066939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:43.066945 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:43.066982 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:43.068593 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.069270 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.069272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.069274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.069380 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.069867 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.069872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.069875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.070097 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:43.070182 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:43.071026 (+ 844us) server_negotiation.cc:299] Negotiation successful
0929 03:10:43.071204 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:10:43.073029 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:43.057931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40787 (local address 127.20.174.65:53851)
0929 03:10:43.058605 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:43.058657 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:43.060066 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:43.060686 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:43.060694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:43.061106 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:43.061631 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:43.061645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.062511 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.062514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:43.063454 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:43.063468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.064099 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.064104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:43.064318 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:43.064978 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:43.064999 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:43.066749 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:43.068722 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:43.068731 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:43.068751 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:43.069155 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:43.069477 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:43.069483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:43.069488 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:43.069700 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:43.070214 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:43.070222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:43.070659 (+ 437us) client_negotiation.cc:769] Sending connection context
0929 03:10:43.070909 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 03:10:43.071585 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:10:43.074895 26952 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40787
I20260929 03:10:43.075205 26952 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:43.075737 26952 heartbeater.cc:511] Master 127.20.174.126:40787 requested a full tablet report, sending...
I20260929 03:10:43.077329 26761 ts_manager.cc:194] Registered new tserver with Master: c318604d679d42f3982f22660895aabd (127.20.174.65:35335)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:10:43.078648 26761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:53851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:10:43.097278 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40787
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:43.212692 26957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:43.212967 26957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:43.213011 26957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:43.217466 26957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:43.217541 26957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:43.217603 26957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:43.223040 26957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:43.224102 26957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:43.225080 26957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:43.235271 26962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:43.235284 26965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:43.235271 26963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:43.235522 26957 server_base.cc:1034] running on GCE node
I20260929 03:10:43.236066 26957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:43.236810 26957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:43.238008 26957 hybrid_clock.cc:648] HybridClock initialized: now 1790651443237992 us; error 40 us; skew 500 ppm
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:43.240962 26957 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:10:43.241927 26957 webserver.cc:460] Webserver started at http://127.20.174.66:38109/ using document root <none> and password file <none>
I20260929 03:10:43.242594 26957 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:43.242692 26957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:43.242935 26957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:43.245268 26957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "b4e5ee59f54a45c590aa7cdb1cd2d332"
format_stamp: "Formatted at 2026-09-29 03:10:43 on dist-test-slave-k1dg"
server_key: "fa9086ef5addc3975257b43e958dcab2"
server_key_iv: "624a96d6f814e830d8ba4ae838827e44"
server_key_version: "encryptionkey@0"
I20260929 03:10:43.245898 26957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "b4e5ee59f54a45c590aa7cdb1cd2d332"
format_stamp: "Formatted at 2026-09-29 03:10:43 on dist-test-slave-k1dg"
server_key: "fa9086ef5addc3975257b43e958dcab2"
server_key_iv: "624a96d6f814e830d8ba4ae838827e44"
server_key_version: "encryptionkey@0"
I20260929 03:10:43.249887 26957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:10:43.252539 26972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:43.253841 26957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:10:43.253985 26957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "b4e5ee59f54a45c590aa7cdb1cd2d332"
format_stamp: "Formatted at 2026-09-29 03:10:43 on dist-test-slave-k1dg"
server_key: "fa9086ef5addc3975257b43e958dcab2"
server_key_iv: "624a96d6f814e830d8ba4ae838827e44"
server_key_version: "encryptionkey@0"
I20260929 03:10:43.254107 26957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:43.274501 26957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:43.278056 26957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:43.278267 26957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:43.278954 26957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:43.279850 26957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:43.279919 26957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:43.279997 26957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:43.280037 26957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:43.289949 26957 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:36219
I20260929 03:10:43.289981 27085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:36219 every 8 connection(s)
I20260929 03:10:43.290949 26957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:43.294117 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 26957
I20260929 03:10:43.294246 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:43.294605 21177 external_mini_cluster.cc:1428] Setting key d0baacc570f7e9bd787d9e14bfa7e098
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:10:43.308575 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:43.292909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:42677 (local address 127.20.174.126:40787)
0929 03:10:43.293125 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:43.293129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:43.294120 (+ 991us) server_negotiation.cc:407] Connection header received
0929 03:10:43.295816 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:43.295819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:43.295864 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:43.295955 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:43.298881 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.299837 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.301407 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.301805 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.301949 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:43.304755 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:43.304758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:43.304760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:43.304787 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:43.306142 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.306802 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.306806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.306808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.306889 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.307324 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.307330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.307335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.307555 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:43.307635 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:43.308336 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 03:10:43.308448 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 03:10:43.309648 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:43.293390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40787 (local address 127.20.174.66:42677)
0929 03:10:43.293913 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:43.294013 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:43.295565 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:43.297684 (+ 2119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:43.297691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:43.298102 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:43.298714 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:43.298732 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.300024 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.300027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:43.301268 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:43.301280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.302003 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.302006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:43.302180 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:43.302851 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:43.302869 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:43.304561 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:43.306308 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:43.306315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:43.306333 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:43.306691 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:43.307014 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:43.307016 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:43.307018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:43.307218 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:43.307663 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:43.307674 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:43.308059 (+ 385us) client_negotiation.cc:769] Sending connection context
0929 03:10:43.308289 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 03:10:43.308697 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260929 03:10:43.310988 27086 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40787
I20260929 03:10:43.311282 27086 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:43.311952 27086 heartbeater.cc:511] Master 127.20.174.126:40787 requested a full tablet report, sending...
I20260929 03:10:43.313192 26761 ts_manager.cc:194] Registered new tserver with Master: b4e5ee59f54a45c590aa7cdb1cd2d332 (127.20.174.66:36219)
I20260929 03:10:43.313812 26761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:42677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:10:43.326642 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40787
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:43.443827 27091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:43.444205 27091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:43.444278 27091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:43.448884 27091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:43.449010 27091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:43.449081 27091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:43.454900 27091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:43.456370 27091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:43.457538 27091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:43.467692 27097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:43.467674 27099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:43.467695 27096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:43.467983 27091 server_base.cc:1034] running on GCE node
I20260929 03:10:43.468501 27091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:43.469174 27091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:43.470364 27091 hybrid_clock.cc:648] HybridClock initialized: now 1790651443470323 us; error 56 us; skew 500 ppm
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:43.473222 27091 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:10:43.474283 27091 webserver.cc:460] Webserver started at http://127.20.174.67:41283/ using document root <none> and password file <none>
I20260929 03:10:43.474928 27091 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:43.475020 27091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:43.475263 27091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:43.477433 27091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "0daafcd0dbc74f2cae4a184623439a2c"
format_stamp: "Formatted at 2026-09-29 03:10:43 on dist-test-slave-k1dg"
server_key: "95213ec1ddf19e8443f79ae4ea4fe4eb"
server_key_iv: "ea177692e73270d256f87ebac0b17743"
server_key_version: "encryptionkey@0"
I20260929 03:10:43.478026 27091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "0daafcd0dbc74f2cae4a184623439a2c"
format_stamp: "Formatted at 2026-09-29 03:10:43 on dist-test-slave-k1dg"
server_key: "95213ec1ddf19e8443f79ae4ea4fe4eb"
server_key_iv: "ea177692e73270d256f87ebac0b17743"
server_key_version: "encryptionkey@0"
I20260929 03:10:43.482051 27091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:10:43.484740 27106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:43.486019 27091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:10:43.486156 27091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "0daafcd0dbc74f2cae4a184623439a2c"
format_stamp: "Formatted at 2026-09-29 03:10:43 on dist-test-slave-k1dg"
server_key: "95213ec1ddf19e8443f79ae4ea4fe4eb"
server_key_iv: "ea177692e73270d256f87ebac0b17743"
server_key_version: "encryptionkey@0"
I20260929 03:10:43.486274 27091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:43.508157 27091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:43.511502 27091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:43.511719 27091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:43.512333 27091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:43.513197 27091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:43.513265 27091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:43.513361 27091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:43.513407 27091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:43.522667 27091 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:36485
I20260929 03:10:43.522687 27219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:36485 every 8 connection(s)
I20260929 03:10:43.523628 27091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:43.533968 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 27091
I20260929 03:10:43.534102 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:43.534395 21177 external_mini_cluster.cc:1428] Setting key bf0b14ebf7dbb4ae69ddb0cec065cec1
I20260929 03:10:43.538770 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:43.525512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:41735 (local address 127.20.174.126:40787)
0929 03:10:43.525662 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:43.525666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:43.526556 (+ 890us) server_negotiation.cc:407] Connection header received
0929 03:10:43.528055 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:43.528058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:43.528121 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:43.528226 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:43.529482 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.530012 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.531243 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.531471 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.531583 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:43.534821 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:43.534825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:43.534827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:43.534854 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:43.536483 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.537083 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.537086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.537087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.537167 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.537492 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.537494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.537495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.537680 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:43.537764 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:43.538359 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 03:10:43.538635 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:10:43.539572 27222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:43.525902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40787 (local address 127.20.174.67:41735)
0929 03:10:43.526366 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:43.526409 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:43.527821 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:43.528429 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:43.528441 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:43.528828 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:43.529326 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:43.529339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.530142 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.530145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:43.531113 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:43.531123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.531625 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.531629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:43.531802 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:43.532482 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:43.532507 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:43.534630 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:43.536654 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:43.536660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:43.536678 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:43.536982 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:43.537243 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:43.537246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:43.537247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:43.537383 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:43.537797 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:43.537802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:43.538059 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 03:10:43.538232 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 03:10:43.538629 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":126,"threads_started":1}
I20260929 03:10:43.540771 27220 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40787
I20260929 03:10:43.541049 27220 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:43.541590 27220 heartbeater.cc:511] Master 127.20.174.126:40787 requested a full tablet report, sending...
I20260929 03:10:43.542774 26761 ts_manager.cc:194] Registered new tserver with Master: 0daafcd0dbc74f2cae4a184623439a2c (127.20.174.67:36485)
I20260929 03:10:43.543376 26761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:41735
I20260929 03:10:43.548390 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/test-admin.keytab.
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:43.565639 21177 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 03:10:43.568739 27227 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:43.567156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.20.174.65:35335)
0929 03:10:43.567538 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:43.567558 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:43.567580 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:10:43.567628 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:43.567633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:43.567835 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:43.567998 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:43.568553 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59016: BlockingRecv error: recv got EOF from 127.0.0.1:59016 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":123,"threads_started":1}
W20260929 03:10:43.568905 21177 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:43.570055 21177 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:43.571070 21177 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:10:43.581169 27227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:43.571983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59020 (local address 127.20.174.65:35335)
0929 03:10:43.572135 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:43.572139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:43.572254 (+ 115us) server_negotiation.cc:407] Connection header received
0929 03:10:43.572413 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:43.572416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:43.572470 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:43.572558 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:43.573119 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.573932 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.574872 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.575283 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.575405 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:43.577728 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:43.577744 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:43.577757 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:43.577783 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:43.579309 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.579797 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.579803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.579804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.579884 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.580187 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.580190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.580192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.580452 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:43.580675 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:43.580860 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 03:10:43.581027 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 03:10:43.585141 27227 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:43.584057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.20.174.65:35335)
0929 03:10:43.584194 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:43.584198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:43.584332 (+ 134us) server_negotiation.cc:407] Connection header received
0929 03:10:43.584481 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:43.584486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:43.584529 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:43.584613 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:43.585023 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59032: BlockingRecv error: recv got EOF from 127.0.0.1:59032 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 03:10:43.585274 21177 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:43.586273 21177 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:43.587185 21177 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:10:43.596112 27227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:43.588112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.20.174.65:35335)
0929 03:10:43.588236 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:43.588240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:43.588352 (+ 112us) server_negotiation.cc:407] Connection header received
0929 03:10:43.588506 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:43.588509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:43.588552 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:43.588632 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:43.589141 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.589647 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.590606 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.590840 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.590964 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:43.593150 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:43.593155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:43.593164 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:43.593187 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:43.594384 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.594870 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.594875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.594877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.594965 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.595240 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.595245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.595246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.595445 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:43.595589 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:43.595824 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 03:10:43.595954 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 03:10:43.599018 27227 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:43.597871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.20.174.65:35335)
0929 03:10:43.598030 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:43.598035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:43.598139 (+ 104us) server_negotiation.cc:407] Connection header received
0929 03:10:43.598281 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:43.598284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:43.598324 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:43.598406 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:43.598888 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59056: BlockingRecv error: recv got EOF from 127.0.0.1:59056 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 03:10:43.599207 21177 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651413504048-21177-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:43.600281 21177 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:43.601158 21177 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[26716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:10:43.610013 27227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:43.602075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.20.174.65:35335)
0929 03:10:43.602186 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:43.602190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:43.602316 (+ 126us) server_negotiation.cc:407] Connection header received
0929 03:10:43.602400 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:43.602402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:43.602531 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:43.602597 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:43.603061 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.603556 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.604332 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:43.604550 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:43.604647 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:43.606998 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:43.607001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:43.607003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:43.607026 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:43.608301 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.608698 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.608701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.608702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.608774 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:43.609087 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:43.609093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:43.609094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:43.609282 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:43.609486 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:43.609649 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 03:10:43.609748 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:10:43.611164 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 26823
I20260929 03:10:43.617057 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 26957
I20260929 03:10:43.622291 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 27091
I20260929 03:10:43.627930 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 26729
2026-09-29T03:10:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2322 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[27240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[27240](info): set up 2 sockets
Sep 29 03:10:43 dist-test-slave-k1dg krb5kdc[27240](info): commencing operation
krb5kdc: starting...
W20260929 03:10:44.667042 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:44 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.126.keytab.
Entry for principal oryx/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.126.keytab.
I20260929 03:10:44.797465 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44627
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.126.keytab
--principal=oryx/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:44627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:44.918515 27253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:44.918893 27253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:44.918989 27253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:44.923422 27253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:44.923525 27253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:44.923561 27253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:44.923604 27253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:44.923640 27253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:44.929195 27253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:44627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.126.keytab
--principal=oryx/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:44.930330 27253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:44.931402 27253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:44.940385 27258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:44.940385 27261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:44.940449 27259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:44.940743 27253 server_base.cc:1034] running on GCE node
I20260929 03:10:44.941517 27253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:44.942813 27253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:44.944082 27253 hybrid_clock.cc:648] HybridClock initialized: now 1790651444944072 us; error 38 us; skew 500 ppm
Sep 29 03:10:44 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651444, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:44.954770 27253 init.cc:377] Logged in from keytab as oryx/127.20.174.126@KRBTEST.COM (short username oryx)
I20260929 03:10:44.955833 27253 webserver.cc:460] Webserver started at http://127.20.174.126:43837/ using document root <none> and password file <none>
I20260929 03:10:44.956459 27253 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:44.956555 27253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:44.956794 27253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:44.959043 27253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "51489f943cd34cbb87a0d04a58ab918c"
format_stamp: "Formatted at 2026-09-29 03:10:44 on dist-test-slave-k1dg"
server_key: "4a60ff8a84af9d3f49c5721987a6cc9e"
server_key_iv: "5bced39a4bb4603c6404e78b116e7525"
server_key_version: "encryptionkey@0"
I20260929 03:10:44.959664 27253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "51489f943cd34cbb87a0d04a58ab918c"
format_stamp: "Formatted at 2026-09-29 03:10:44 on dist-test-slave-k1dg"
server_key: "4a60ff8a84af9d3f49c5721987a6cc9e"
server_key_iv: "5bced39a4bb4603c6404e78b116e7525"
server_key_version: "encryptionkey@0"
I20260929 03:10:44.963732 27253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 03:10:44.966360 27268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:44.968098 27253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:10:44.968302 27253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "51489f943cd34cbb87a0d04a58ab918c"
format_stamp: "Formatted at 2026-09-29 03:10:44 on dist-test-slave-k1dg"
server_key: "4a60ff8a84af9d3f49c5721987a6cc9e"
server_key_iv: "5bced39a4bb4603c6404e78b116e7525"
server_key_version: "encryptionkey@0"
I20260929 03:10:44.968451 27253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:44.993461 27253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:44.997123 27253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:44.997344 27253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:45.004915 27253 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:44627
I20260929 03:10:45.005317 27320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:44627 every 8 connection(s)
I20260929 03:10:45.006125 27253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:45.009346 27321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:45.014689 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 27253
I20260929 03:10:45.014858 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:45.015202 21177 external_mini_cluster.cc:1428] Setting key 604ad5a0ae85b71563ef5833ad8ce6b4
I20260929 03:10:45.016494 27321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c: Bootstrap starting.
I20260929 03:10:45.019244 27321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:45.020265 27321 log.cc:824] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:45.022737 27321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c: No bootstrap required, opened a new log
Sep 29 03:10:45 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:45.028384 27321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } }
I20260929 03:10:45.028716 27321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:45.028785 27321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51489f943cd34cbb87a0d04a58ab918c, State: Initialized, Role: FOLLOWER
I20260929 03:10:45.029392 27321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } }
I20260929 03:10:45.029609 27321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:45.029690 27321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:45.029798 27321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:45.031164 27321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } }
I20260929 03:10:45.031570 27321 leader_election.cc:302] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51489f943cd34cbb87a0d04a58ab918c; no voters:
I20260929 03:10:45.031948 27321 leader_election.cc:288] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:45.032011 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:45.016842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.20.174.126:44627)
0929 03:10:45.017293 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:45.017319 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:45.017358 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:10:45.018051 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:45.018070 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:45.018489 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:45.018811 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:45.019880 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.021018 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.022271 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.022816 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.022994 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:45.025975 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:45.025982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:45.026003 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:45.026047 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:45.028353 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:45.028865 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:45.028873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:45.028883 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:45.029039 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:45.029369 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:45.029372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:45.029374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:45.030029 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:45.030234 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:45.030524 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 03:10:45.030875 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":168,"threads_started":1}
I20260929 03:10:45.032104 27326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:45.032377 27326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 1 LEADER]: Becoming Leader. State: Replica: 51489f943cd34cbb87a0d04a58ab918c, State: Running, Role: LEADER
I20260929 03:10:45.032739 27326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } }
I20260929 03:10:45.033985 27326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51489f943cd34cbb87a0d04a58ab918c. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } }
I20260929 03:10:45.034190 27326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:45.034554 27321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:45.034520 27328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } }
I20260929 03:10:45.034680 27328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:45.035260 27330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:45.038653 27330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:45.044610 27330 catalog_manager.cc:1347] Generated new cluster ID: f9cd3cde7b3b45f2a0a453c7770a66f0
I20260929 03:10:45.044695 27330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:45.052549 27330 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:45.053580 27330 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:45.060021 27330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c: Generated new TSK 0
I20260929 03:10:45.060804 27330 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab.
Entry for principal oryx/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab.
I20260929 03:10:45.094022 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44627
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab
--principal=oryx/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:45.209901 27347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:45.210191 27347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:45.210234 27347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:45.215538 27347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:45.215871 27347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:45.216022 27347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:45.225132 27347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab
--principal=oryx/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:45.226780 27347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:45.228762 27347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:45.241576 27353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:45.241576 27352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:45.241770 27347 server_base.cc:1034] running on GCE node
W20260929 03:10:45.242012 27355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:45.242583 27347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:45.243570 27347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:45.244716 27347 hybrid_clock.cc:648] HybridClock initialized: now 1790651445244701 us; error 33 us; skew 500 ppm
Sep 29 03:10:45 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651445, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:45.248704 27347 init.cc:377] Logged in from keytab as oryx/127.20.174.65@KRBTEST.COM (short username oryx)
I20260929 03:10:45.249930 27347 webserver.cc:460] Webserver started at http://127.20.174.65:41333/ using document root <none> and password file <none>
I20260929 03:10:45.250797 27347 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:45.250910 27347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:45.251195 27347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:45.253536 27347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "23541f7f22ce42debe52d0985731995b"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "a6c0aec3b74042b6e4860f041fecc011"
server_key_iv: "5faf04f0bf089c1214ce1f931fb22a39"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.254119 27347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "23541f7f22ce42debe52d0985731995b"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "a6c0aec3b74042b6e4860f041fecc011"
server_key_iv: "5faf04f0bf089c1214ce1f931fb22a39"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.258386 27347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:10:45.261409 27362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:45.263001 27347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:45.263159 27347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "23541f7f22ce42debe52d0985731995b"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "a6c0aec3b74042b6e4860f041fecc011"
server_key_iv: "5faf04f0bf089c1214ce1f931fb22a39"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.263293 27347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:45.277066 27347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:45.280550 27347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:45.280774 27347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:45.281352 27347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:45.282167 27347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:45.282214 27347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:45.282256 27347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:45.282279 27347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:45.291656 27347 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:44577
I20260929 03:10:45.291674 27475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:44577 every 8 connection(s)
I20260929 03:10:45.292574 27347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:45.301225 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 27347
I20260929 03:10:45.301376 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:45.301712 21177 external_mini_cluster.cc:1428] Setting key 8cea84e99d6a689cceac252e35c6ea3b
Sep 29 03:10:45 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651445, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:45.308171 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:45.294499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:55237 (local address 127.20.174.126:44627)
0929 03:10:45.294636 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:45.294640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:45.295536 (+ 896us) server_negotiation.cc:407] Connection header received
0929 03:10:45.296937 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:45.296940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:45.296997 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:45.297073 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:45.298387 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.299000 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.300213 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.300440 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.300518 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:45.303557 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:45.303562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:45.303565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:45.303592 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:45.305452 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:45.306192 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:45.306195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:45.306197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:45.306293 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:45.306845 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:45.306848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:45.306849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:45.307083 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:45.307187 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:45.307933 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 03:10:45.308049 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:10:45.310153 27478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:45.294900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.65:55237)
0929 03:10:45.295371 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:45.295416 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:45.296684 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:45.297294 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:45.297302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:45.297679 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:45.298219 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:45.298233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.299144 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.299147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:45.300091 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:45.300103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.300575 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.300578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:45.300728 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:45.301527 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:45.301545 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:45.303385 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:45.305597 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:45.305606 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:45.305627 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:45.306066 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:45.306396 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:45.306401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:45.306406 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:45.306650 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:45.307212 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:45.307221 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:45.307592 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 03:10:45.307817 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 03:10:45.308211 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20260929 03:10:45.311597 27476 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44627
I20260929 03:10:45.311892 27476 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:45.312587 27476 heartbeater.cc:511] Master 127.20.174.126:44627 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:10:45.314342 27285 ts_manager.cc:194] Registered new tserver with Master: 23541f7f22ce42debe52d0985731995b (127.20.174.65:44577)
I20260929 03:10:45.315593 27285 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:55237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab.
Entry for principal oryx/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab.
I20260929 03:10:45.333523 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44627
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab
--principal=oryx/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:45.452054 27481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:45.452534 27481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:45.452632 27481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:45.456907 27481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:45.456971 27481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:45.457072 27481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:45.462863 27481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab
--principal=oryx/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:45.463996 27481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:45.465075 27481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:10:45.475914 27481 server_base.cc:1034] running on GCE node
W20260929 03:10:45.475973 27486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:45.475958 27487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:45.476172 27489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:45.476642 27481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:45.477257 27481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:45.478503 27481 hybrid_clock.cc:648] HybridClock initialized: now 1790651445478473 us; error 41 us; skew 500 ppm
Sep 29 03:10:45 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651445, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:45.481392 27481 init.cc:377] Logged in from keytab as oryx/127.20.174.66@KRBTEST.COM (short username oryx)
I20260929 03:10:45.482381 27481 webserver.cc:460] Webserver started at http://127.20.174.66:37027/ using document root <none> and password file <none>
I20260929 03:10:45.483033 27481 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:45.483100 27481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:45.483345 27481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:45.485541 27481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "b0c90b2b03d847c8a4a68151a867bb2e"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "6a69e08cfa68a0a67a6b18dbd959eece"
server_key_iv: "75f68f710fcf3a2c62073ca13933b69c"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.486135 27481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "b0c90b2b03d847c8a4a68151a867bb2e"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "6a69e08cfa68a0a67a6b18dbd959eece"
server_key_iv: "75f68f710fcf3a2c62073ca13933b69c"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.490233 27481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 03:10:45.493069 27496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:45.494613 27481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:45.494753 27481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "b0c90b2b03d847c8a4a68151a867bb2e"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "6a69e08cfa68a0a67a6b18dbd959eece"
server_key_iv: "75f68f710fcf3a2c62073ca13933b69c"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.494881 27481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:45.509162 27481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:45.512584 27481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:45.512799 27481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:45.513499 27481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:45.514668 27481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:45.514743 27481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:45.514809 27481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:45.514833 27481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:45.523960 27481 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:35397
I20260929 03:10:45.524001 27609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:35397 every 8 connection(s)
I20260929 03:10:45.524906 27481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:45.530817 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 27481
I20260929 03:10:45.530990 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:45.531344 21177 external_mini_cluster.cc:1428] Setting key 4043caa6d0428a8c504132f1f373c4e4
Sep 29 03:10:45 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651445, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:45.541121 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:45.526877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:44971 (local address 127.20.174.126:44627)
0929 03:10:45.527034 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:45.527037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:45.528088 (+ 1051us) server_negotiation.cc:407] Connection header received
0929 03:10:45.529508 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:45.529512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:45.529574 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:45.529656 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:45.531246 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.531932 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.533507 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.533821 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.533913 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:45.537242 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:45.537246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:45.537249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:45.537282 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:45.538988 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:45.539538 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:45.539542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:45.539544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:45.539629 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:45.539984 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:45.539988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:45.539989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:45.540182 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:45.540290 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:45.540811 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 03:10:45.540942 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for oryx/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:10:45.542059 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:45.527383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.66:44971)
0929 03:10:45.527921 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:45.527970 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:45.529264 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:45.529843 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:45.529851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:45.530242 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:45.531025 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:45.531050 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.532063 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.532068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:45.533370 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:45.533386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.533998 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.534004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:45.534248 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:45.534991 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:45.535015 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:45.537053 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:45.539114 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:45.539120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:45.539137 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:45.539419 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:45.539725 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:45.539728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:45.539730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:45.539882 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:45.540289 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:45.540297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:45.540569 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 03:10:45.540771 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 03:10:45.541143 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
I20260929 03:10:45.543772 27610 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44627
I20260929 03:10:45.544018 27610 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:45.544627 27610 heartbeater.cc:511] Master 127.20.174.126:44627 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.67@KRBTEST.COM" created.
I20260929 03:10:45.545859 27285 ts_manager.cc:194] Registered new tserver with Master: b0c90b2b03d847c8a4a68151a867bb2e (127.20.174.66:35397)
I20260929 03:10:45.546554 27285 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:44971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab.
Entry for principal oryx/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab.
I20260929 03:10:45.562628 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44627
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab
--principal=oryx/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:45.681834 27615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:45.682260 27615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:45.682360 27615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:45.688556 27615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:45.688663 27615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:45.688771 27615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:45.694845 27615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab
--principal=oryx/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:45.695995 27615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:45.697047 27615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:10:45.707607 27615 server_base.cc:1034] running on GCE node
W20260929 03:10:45.707554 27623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:45.707554 27621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:45.707572 27620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:45.708271 27615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:45.708961 27615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:45.710125 27615 hybrid_clock.cc:648] HybridClock initialized: now 1790651445710103 us; error 36 us; skew 500 ppm
Sep 29 03:10:45 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651445, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:45.713068 27615 init.cc:377] Logged in from keytab as oryx/127.20.174.67@KRBTEST.COM (short username oryx)
I20260929 03:10:45.714004 27615 webserver.cc:460] Webserver started at http://127.20.174.67:43513/ using document root <none> and password file <none>
I20260929 03:10:45.714612 27615 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:45.714679 27615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:45.714855 27615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:45.716908 27615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "1aad6b620fd141db897db0570f975caf"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "8c4fd9e207746c101125bf8e8fa08d09"
server_key_iv: "ee97e29348cd6d97a3ff6f35f4e0f39f"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.717442 27615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "1aad6b620fd141db897db0570f975caf"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "8c4fd9e207746c101125bf8e8fa08d09"
server_key_iv: "ee97e29348cd6d97a3ff6f35f4e0f39f"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.721318 27615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 03:10:45.723852 27630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:45.725138 27615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:10:45.725237 27615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "1aad6b620fd141db897db0570f975caf"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "8c4fd9e207746c101125bf8e8fa08d09"
server_key_iv: "ee97e29348cd6d97a3ff6f35f4e0f39f"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.725328 27615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:45.755781 27615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:45.759397 27615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:45.759563 27615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:45.760146 27615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:45.760999 27615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:45.761049 27615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:45.761092 27615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:45.761113 27615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:45.770298 27615 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:35159
I20260929 03:10:45.770320 27743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:35159 every 8 connection(s)
I20260929 03:10:45.771286 27615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:45.779671 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 27615
I20260929 03:10:45.779824 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:45.780144 21177 external_mini_cluster.cc:1428] Setting key a665f3c82d5e463a3b0f95a4a58aa723
Sep 29 03:10:45 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651445, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:45.785643 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:45.773275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:54367 (local address 127.20.174.126:44627)
0929 03:10:45.773407 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:45.773411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:45.774284 (+ 873us) server_negotiation.cc:407] Connection header received
0929 03:10:45.775721 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:45.775724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:45.775787 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:45.775885 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:45.777076 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.777670 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.779289 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.779632 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.779755 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:45.782612 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:45.782615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:45.782617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:45.782643 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:45.783840 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:45.784329 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:45.784332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:45.784333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:45.784408 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:45.784696 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:45.784698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:45.784700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:45.784885 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:45.784982 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:45.785417 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 03:10:45.785522 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:10:45.786834 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:45.773677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.67:54367)
0929 03:10:45.774125 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:45.774168 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:45.775494 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:45.776065 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:45.776072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:45.776442 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:45.776930 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:45.776943 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.777795 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.777801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:45.779159 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:45.779170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:45.779856 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:45.779863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:45.780051 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:45.780727 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:45.780749 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:45.782420 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:45.783952 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:45.783957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:45.783971 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:45.784232 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:45.784481 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:45.784483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:45.784484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:45.784616 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:45.784966 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:45.784971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:45.785207 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 03:10:45.785356 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 03:10:45.785638 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
I20260929 03:10:45.788091 27744 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44627
I20260929 03:10:45.788349 27744 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:45.788868 27744 heartbeater.cc:511] Master 127.20.174.126:44627 requested a full tablet report, sending...
I20260929 03:10:45.789862 27285 ts_manager.cc:194] Registered new tserver with Master: 1aad6b620fd141db897db0570f975caf (127.20.174.67:35159)
I20260929 03:10:45.790621 27285 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:54367
I20260929 03:10:45.794250 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.174.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.125.keytab.
Entry for principal oryx/127.20.174.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.125.keytab.
I20260929 03:10:45.823982 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.125:43575
--webserver_interface=127.20.174.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.125.keytab
--principal=oryx/127.20.174.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:44627,127.20.174.125:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:45.942986 27749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:45.943298 27749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:45.943346 27749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:45.947885 27749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:45.947957 27749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:45.947978 27749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:45.947994 27749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:45.948010 27749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:45.953405 27749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:44627,127.20.174.125:43575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.125.keytab
--principal=oryx/127.20.174.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.125:43575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.174.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
W20260929 03:10:45.953619 27749 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.174.126:44627,127.20.174.125:43575'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 03:10:45.954586 27749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:45.955607 27749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:45.964408 27754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:45.964416 27755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:45.964430 27757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:45.964679 27749 server_base.cc:1034] running on GCE node
I20260929 03:10:45.965428 27749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:45.966673 27749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:45.967839 27749 hybrid_clock.cc:648] HybridClock initialized: now 1790651445967801 us; error 53 us; skew 500 ppm
Sep 29 03:10:45 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651445, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:45.970853 27749 init.cc:377] Logged in from keytab as oryx/127.20.174.125@KRBTEST.COM (short username oryx)
I20260929 03:10:45.971845 27749 webserver.cc:460] Webserver started at http://127.20.174.125:39905/ using document root <none> and password file <none>
I20260929 03:10:45.972537 27749 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:45.972630 27749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:45.972852 27749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:45.975253 27749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data/instance:
uuid: "573016e12e7e4aa9b3e8161371367e69"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "811ec0663f014cbf8d59520599605288"
server_key_iv: "059e2d3aa6ed190fd669e29ce5c82dc8"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.975872 27749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/wal/instance:
uuid: "573016e12e7e4aa9b3e8161371367e69"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "811ec0663f014cbf8d59520599605288"
server_key_iv: "059e2d3aa6ed190fd669e29ce5c82dc8"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.979941 27749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:10:45.982645 27764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:45.984027 27749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:10:45.984294 27749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/wal
uuid: "573016e12e7e4aa9b3e8161371367e69"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "811ec0663f014cbf8d59520599605288"
server_key_iv: "059e2d3aa6ed190fd669e29ce5c82dc8"
server_key_version: "encryptionkey@0"
I20260929 03:10:45.984439 27749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:45.998641 27749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:46.002250 27749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:46.002534 27749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:46.010742 27749 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.125:43575
I20260929 03:10:46.011055 27816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.125:43575 every 8 connection(s)
I20260929 03:10:46.011888 27749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
I20260929 03:10:46.014848 27817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:46.019212 27817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:10:46.020761 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 27749
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651445, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.125@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.174.125@KRBTEST.COM
I20260929 03:10:46.035528 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.021292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.20.174.126:44627)
0929 03:10:46.021480 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.021485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.022846 (+ 1361us) server_negotiation.cc:407] Connection header received
0929 03:10:46.023350 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.023353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.023399 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.023511 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.024680 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.025490 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.026940 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.027280 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.027384 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.030938 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.030942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.030945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.030996 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.032878 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.033514 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.033520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.033523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.033619 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.034064 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.034067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.034069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.034344 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.034503 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.035205 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.035338 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:10:46.036491 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.021764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:43774)
0929 03:10:46.022702 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.022757 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.023156 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.023618 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.023624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.024172 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.024545 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.024559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.025624 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.025626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.026825 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.026835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.027438 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.027440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.027656 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.028406 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.028427 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.030735 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.032994 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.033002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.033010 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.033390 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.033730 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.033733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.033735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.033948 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.034511 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.034517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.034922 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.035121 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.035553 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":67,"spinlock_wait_cycles":35072,"thread_start_us":182,"threads_started":1}
I20260929 03:10:46.036526 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.021764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.20.174.125:43575)
0929 03:10:46.022295 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.022309 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.022342 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:10:46.023130 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.023149 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.023386 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.023560 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.024480 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.025767 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.026966 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.027521 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.027631 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.031390 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.031396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.031407 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.031447 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.033476 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.034026 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.034033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.034037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.034130 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.034422 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.034427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.034449 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.034803 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.035011 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.035205 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.035757 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":170,"server-negotiator.queue_time_us":404,"spinlock_wait_cycles":8960,"thread_start_us":133,"threads_started":1}
I20260929 03:10:46.038933 27817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651445, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.125@KRBTEST.COM for oryx/127.20.174.125@KRBTEST.COM
I20260929 03:10:46.047277 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.039441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:43575 (local address 127.0.0.1:33800)
0929 03:10:46.039597 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.039610 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.039687 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.039906 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.039908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.040073 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.040161 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.040166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.041223 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.041229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.041743 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.041750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.042169 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.042172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.042351 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.042853 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.042872 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.044149 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.045796 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.045802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.045804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.045993 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.046242 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.046246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.046248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.046324 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.046708 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.046711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.046797 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.046904 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.047089 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":54}
I20260929 03:10:46.047277 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.039523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.20.174.125:43575)
0929 03:10:46.039637 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.039643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.039658 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:46.039778 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.039782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.039822 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.039897 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.040261 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.041121 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.041848 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.042071 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.042172 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.044283 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.044286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.044289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.044315 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.045679 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.046082 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.046088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.046089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.046158 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.046406 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.046417 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.046419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.046613 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.046709 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.046949 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.047150 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":53}
I20260929 03:10:46.050362 27817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: Bootstrap starting.
I20260929 03:10:46.052788 27817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:46.053545 27817 log.cc:824] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:46.055662 27817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: No bootstrap required, opened a new log
I20260929 03:10:46.058830 27817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } }
I20260929 03:10:46.059067 27817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:46.059142 27817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573016e12e7e4aa9b3e8161371367e69, State: Initialized, Role: FOLLOWER
I20260929 03:10:46.059624 27817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } }
I20260929 03:10:46.060140 27823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } } }
I20260929 03:10:46.060285 27823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:10:46.061002 27817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:10:46.065834 27834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:10:46.065944 27834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:10:46.067938 27749 master_runner.cc:428] Detected that this master 573016e12e7e4aa9b3e8161371367e69 is joining an existing cluster
I20260929 03:10:46.068029 27749 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.174.125:43575
I20260929 03:10:46.079738 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.070550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.20.174.125:43575)
0929 03:10:46.070675 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.070682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.071466 (+ 784us) server_negotiation.cc:407] Connection header received
0929 03:10:46.071613 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.071647 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.071691 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.071786 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.072839 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.073655 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.074693 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.075093 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.075212 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.076459 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.076464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.076469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.076495 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.078057 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.078510 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.078513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.078515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.078588 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.078843 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.078846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.078848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.079027 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.079132 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.079434 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.079589 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":44}
I20260929 03:10:46.079749 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.070745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:43575 (local address 127.0.0.1:33806)
0929 03:10:46.071396 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.071408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.071514 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.071946 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.071949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.072623 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.072725 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.072731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.073777 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.073781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.074595 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.074602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.075235 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.075239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.075338 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.075803 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.075818 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.076314 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.078183 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.078186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.078188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.078377 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.078672 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.078677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.078679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.078759 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.079166 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.079169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.079274 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.079382 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.079539 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":71,"threads_started":1}
I20260929 03:10:46.080152 27841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.070550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:43788)
0929 03:10:46.071028 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.071043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.071142 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.072020 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.072025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.072318 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.072474 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.072481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.073332 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.073338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.074188 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.074198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.074833 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.074836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.075020 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.076059 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.076070 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.076445 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.078525 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.078528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.078532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.078773 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.079101 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.079106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.079108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.079188 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.079644 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.079647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.079731 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.079824 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.079970 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":33,"thread_start_us":120,"threads_started":1}
I20260929 03:10:46.080686 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.070649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.20.174.126:44627)
0929 03:10:46.071661 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.071667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.071681 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:10:46.071724 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.071727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.071780 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.071849 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.072600 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.073223 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.074323 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.074715 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.074835 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.076587 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.076591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.076594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.076624 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.078357 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.078896 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.078899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.078901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.079003 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.079285 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.079288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.079290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.079506 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.079599 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.080405 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.080528 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651444, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.126@KRBTEST.COM for oryx/127.20.174.125@KRBTEST.COM
I20260929 03:10:46.095391 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.084841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.20.174.125:43575)
0929 03:10:46.084977 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.084982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.085376 (+ 394us) server_negotiation.cc:407] Connection header received
0929 03:10:46.085552 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.085555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.085596 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.085659 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.086488 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.086957 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.088214 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.088494 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.088573 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.091716 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.091720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.091723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.091751 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.093481 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.093957 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.093960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.093962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.094049 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.094300 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.094304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.094308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.094580 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.094684 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.095103 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.095224 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:10:46.095548 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.084810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:43575 (local address 127.0.0.1:33822)
0929 03:10:46.085231 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.085294 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.085431 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.085716 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.085722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.086177 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.086318 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.086326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.087060 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.087063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.088106 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.088118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.088603 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.088607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.088730 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.089530 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.089545 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.091540 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.093600 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.093609 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.093614 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.093856 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.094139 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.094142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.094143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.094217 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.094697 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.094707 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.094863 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.095039 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.095273 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":121,"threads_started":1}
I20260929 03:10:46.095902 27823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:46.096104 27823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } }
I20260929 03:10:46.097208 27823 leader_election.cc:288] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627)
I20260929 03:10:46.097755 27292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "573016e12e7e4aa9b3e8161371367e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51489f943cd34cbb87a0d04a58ab918c" is_pre_election: true
I20260929 03:10:46.098594 27767 leader_election.cc:302] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 573016e12e7e4aa9b3e8161371367e69; no voters: 51489f943cd34cbb87a0d04a58ab918c
I20260929 03:10:46.098948 27823 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 03:10:46.099061 27285 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.174.125:43575: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "573016e12e7e4aa9b3e8161371367e69"
member_type: NON_VOTER
last_known_addr {
host: "127.20.174.125"
port: 43575
}
attrs {
promote: true
}
}
dest_uuid: "51489f943cd34cbb87a0d04a58ab918c"
cas_config_opid_index: -1
I20260929 03:10:46.099727 27285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } }
I20260929 03:10:46.102049 27791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:46.101500 27846 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 573016e12e7e4aa9b3e8161371367e69 (127.20.174.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.20.174.124@KRBTEST.COM; defaulting to no policy
I20260929 03:10:46.103677 27791 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 FOLLOWER]: Refusing update from remote peer 51489f943cd34cbb87a0d04a58ab918c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 03:10:46.104314 27845 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [LEADER]: Connected to new peer: Peer: permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:46.105518 27846 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.174.125:43575
I20260929 03:10:46.105994 27749 master.cc:537] Master@127.20.174.125:43575 shutting down...
I20260929 03:10:46.105985 27845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } }
I20260929 03:10:46.106084 27845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.124@KRBTEST.COM" created.
I20260929 03:10:46.111702 27791 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 573016e12e7e4aa9b3e8161371367e69 (127.20.174.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } }
I20260929 03:10:46.112941 27850 mvcc.cc:205] Tried to move back new op lower bound from 7334508318961127424 to 7334508318855118848. Current Snapshot: MvccSnapshot[applied={T|T < 7334508318891192320}]
I20260929 03:10:46.113878 27823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51489f943cd34cbb87a0d04a58ab918c. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } }
I20260929 03:10:46.113996 27823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:10:46.114091 27823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } }
I20260929 03:10:46.114162 27823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:10:46.114326 27852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } }
I20260929 03:10:46.114404 27852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:10:46.115247 27749 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 LEARNER]: Raft consensus shutting down.
I20260929 03:10:46.115572 27749 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 LEARNER]: Raft consensus is shut down!
I20260929 03:10:46.115669 27749 tablet_replica.cc:330] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: stopping tablet replica
I20260929 03:10:46.131157 27749 master.cc:552] Master@127.20.174.125:43575 shutdown complete.
I20260929 03:10:46.131222 27749 master_runner.cc:305] Clearing existing system tablet
I20260929 03:10:46.131471 27749 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:46.131532 27749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.124.keytab.
Entry for principal oryx/127.20.174.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.124.keytab.
I20260929 03:10:46.133438 27749 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:10:46.134009 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.124:34329
--webserver_interface=127.20.174.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.124.keytab
--principal=oryx/127.20.174.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:44627,127.20.174.125:43575,127.20.174.124:34329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:10:46.134647 27858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:46.134868 27749 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:46.135030 27749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:46.135078 27749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/wal
uuid: "573016e12e7e4aa9b3e8161371367e69"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "811ec0663f014cbf8d59520599605288"
server_key_iv: "059e2d3aa6ed190fd669e29ce5c82dc8"
server_key_version: "encryptionkey@0"
I20260929 03:10:46.137029 27749 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:46.139464 27749 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:10:46.139562 27749 log.cc:1196] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 03:10:46.139858 27749 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: Deleting consensus metadata
I20260929 03:10:46.140084 27749 master_runner.cc:315] Copying system tablet from 127.20.174.126:44627
I20260929 03:10:46.141299 27749 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: tablet copy: Beginning tablet copy session from remote peer at address 127.20.174.126:44627
I20260929 03:10:46.152045 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.142911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.20.174.126:44627)
0929 03:10:46.143066 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.143071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.143195 (+ 124us) server_negotiation.cc:407] Connection header received
0929 03:10:46.143424 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.143427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.143482 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.143574 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.144201 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.145019 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.146244 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.146615 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.146724 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.148121 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.148125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.148128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.148155 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.150084 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.150738 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.150742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.150744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.150840 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.151139 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.151142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.151144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.151365 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.151463 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.151757 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.151879 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:10:46.152060 27865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.142802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:43800)
0929 03:10:46.143087 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.143105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.143293 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.143613 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.143619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.143861 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.144055 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.144065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.145146 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.145150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.146105 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.146117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.146735 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.146739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.146841 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.147475 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.147489 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.147946 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.150303 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.150307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.150310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.150619 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.150937 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.150941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.150943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.151048 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.151463 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.151467 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.151572 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.151681 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.151837 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
I20260929 03:10:46.153059 27305 tablet_copy_service.cc:140] P 51489f943cd34cbb87a0d04a58ab918c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 573016e12e7e4aa9b3e8161371367e69 ({username='oryx', principal='oryx/127.20.174.125@KRBTEST.COM'} at 127.0.0.1:43800)
I20260929 03:10:46.153203 27305 tablet_copy_service.cc:161] P 51489f943cd34cbb87a0d04a58ab918c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 573016e12e7e4aa9b3e8161371367e69 at {username='oryx', principal='oryx/127.20.174.125@KRBTEST.COM'} at 127.0.0.1:43800: session id = 573016e12e7e4aa9b3e8161371367e69-00000000000000000000000000000000
I20260929 03:10:46.155032 27305 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:10:46.156749 27749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:10:46.159193 27749 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: tablet copy: Starting download of 0 data blocks...
I20260929 03:10:46.159400 27749 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: tablet copy: Starting download of 1 WAL segments...
I20260929 03:10:46.161746 27749 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:10:46.164111 27305 tablet_copy_service.cc:342] P 51489f943cd34cbb87a0d04a58ab918c: Request end of tablet copy session 573016e12e7e4aa9b3e8161371367e69-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.174.125@KRBTEST.COM'} at 127.0.0.1:43800
I20260929 03:10:46.164227 27305 tablet_copy_service.cc:433] P 51489f943cd34cbb87a0d04a58ab918c: ending tablet copy session 573016e12e7e4aa9b3e8161371367e69-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 573016e12e7e4aa9b3e8161371367e69
I20260929 03:10:46.171785 27749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:46.174055 27866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:46.174062 27867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:46.174326 27869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:46.178728 27749 server_base.cc:1034] running on GCE node
I20260929 03:10:46.178951 27749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:46.179287 27749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:46.180439 27749 hybrid_clock.cc:648] HybridClock initialized: now 1790651446180420 us; error 39 us; skew 500 ppm
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:46.182130 27749 init.cc:377] Logged in from keytab as oryx/127.20.174.125@KRBTEST.COM (short username oryx)
I20260929 03:10:46.182710 27749 webserver.cc:460] Webserver started at http://127.20.174.125:33671/ using document root <none> and password file <none>
I20260929 03:10:46.182924 27749 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:46.182971 27749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:46.184399 27749 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:46.185433 27876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:46.185716 27749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:46.185799 27749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/wal
uuid: "573016e12e7e4aa9b3e8161371367e69"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "811ec0663f014cbf8d59520599605288"
server_key_iv: "059e2d3aa6ed190fd669e29ce5c82dc8"
server_key_version: "encryptionkey@0"
I20260929 03:10:46.185895 27749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:46.199869 27749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:46.202193 27749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:46.207309 27749 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.125:43575
I20260929 03:10:46.207345 27928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.125:43575 every 8 connection(s)
I20260929 03:10:46.207923 27749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
I20260929 03:10:46.209847 27929 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:10:46.210994 27929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: Bootstrap starting.
I20260929 03:10:46.218730 27929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:10:46.219059 27929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: Bootstrap complete.
I20260929 03:10:46.219656 27929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } }
I20260929 03:10:46.219808 27929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 573016e12e7e4aa9b3e8161371367e69, State: Initialized, Role: LEARNER
I20260929 03:10:46.219916 27929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } }
I20260929 03:10:46.220292 27932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } }
I20260929 03:10:46.220398 27929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:46.220502 27932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:10:46.223798 27943 catalog_manager.cc:1259] Loaded cluster ID: f9cd3cde7b3b45f2a0a453c7770a66f0
I20260929 03:10:46.223932 27943 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: loading cluster ID for follower catalog manager: success
I20260929 03:10:46.226052 27943 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: acquiring CA information for follower catalog manager: success
I20260929 03:10:46.227449 27943 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:10:46.273201 27859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:46.273518 27859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:46.273604 27859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:46.277904 27859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:46.278002 27859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:46.278038 27859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:46.278074 27859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:46.278105 27859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:46.283659 27859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:44627,127.20.174.125:43575,127.20.174.124:34329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.124.keytab
--principal=oryx/127.20.174.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.124:34329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.174.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:46.284775 27859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:46.285872 27859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:46.294595 27949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:46.294606 27951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:46.294610 27948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:46.294835 27859 server_base.cc:1034] running on GCE node
I20260929 03:10:46.295571 27859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:46.296842 27859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:46.298012 27859 hybrid_clock.cc:648] HybridClock initialized: now 1790651446297992 us; error 34 us; skew 500 ppm
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:46.300819 27859 init.cc:377] Logged in from keytab as oryx/127.20.174.124@KRBTEST.COM (short username oryx)
I20260929 03:10:46.301807 27859 webserver.cc:460] Webserver started at http://127.20.174.124:40407/ using document root <none> and password file <none>
I20260929 03:10:46.302371 27859 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:46.302475 27859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:46.302699 27859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:46.304776 27859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data/instance:
uuid: "d2be03d77f4340b79c5056a541959897"
format_stamp: "Formatted at 2026-09-29 03:10:46 on dist-test-slave-k1dg"
server_key: "c44cf2d6c01a7b209ba18b533c203a82"
server_key_iv: "b677d7f69c258e4e8601e727659d8566"
server_key_version: "encryptionkey@0"
I20260929 03:10:46.305347 27859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/wal/instance:
uuid: "d2be03d77f4340b79c5056a541959897"
format_stamp: "Formatted at 2026-09-29 03:10:46 on dist-test-slave-k1dg"
server_key: "c44cf2d6c01a7b209ba18b533c203a82"
server_key_iv: "b677d7f69c258e4e8601e727659d8566"
server_key_version: "encryptionkey@0"
I20260929 03:10:46.309324 27859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 03:10:46.311856 27958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:46.313210 27859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:10:46.313375 27859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/wal
uuid: "d2be03d77f4340b79c5056a541959897"
format_stamp: "Formatted at 2026-09-29 03:10:46 on dist-test-slave-k1dg"
server_key: "c44cf2d6c01a7b209ba18b533c203a82"
server_key_iv: "b677d7f69c258e4e8601e727659d8566"
server_key_version: "encryptionkey@0"
I20260929 03:10:46.313524 27859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:46.318296 27476 heartbeater.cc:503] Master 127.20.174.126:44627 was elected leader, sending a full tablet report...
I20260929 03:10:46.338426 27859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:46.341944 27859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:46.342191 27859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:46.350419 27859 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.124:34329
I20260929 03:10:46.350751 28010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.124:34329 every 8 connection(s)
I20260929 03:10:46.351684 27859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
I20260929 03:10:46.354602 28011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:46.358560 28011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:10:46.361529 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 27859
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.124@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.174.124@KRBTEST.COM
I20260929 03:10:46.374162 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.360596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.20.174.126:44627)
0929 03:10:46.360739 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.360743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.361692 (+ 949us) server_negotiation.cc:407] Connection header received
0929 03:10:46.363087 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.363090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.363140 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.363209 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.364631 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.365176 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.366404 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.366687 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.366771 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.369802 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.369808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.369812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.369849 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.371530 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.372170 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.372177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.372179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.372260 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.372743 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.372747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.372748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.373011 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.373086 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.373837 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.373980 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:10:46.375445 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.362615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.20.174.124:34329)
0929 03:10:46.363006 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.363020 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.363061 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:10:46.363253 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.363259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.363499 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.363695 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.364373 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.365554 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.366406 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.366938 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.367042 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.370256 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.370264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.370282 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.370318 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.372469 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.372994 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.372999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.373000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.373108 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.373398 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.373401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.373403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.373682 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.373962 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.374118 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.374253 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":282,"spinlock_wait_cycles":1408,"thread_start_us":183,"threads_started":1}
I20260929 03:10:46.375445 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.360896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:43810)
0929 03:10:46.361493 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.361564 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.362803 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.363304 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.363313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.363805 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.364477 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.364492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.365322 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.365326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.366269 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.366279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.366835 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.366839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.367036 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.367701 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.367730 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.369563 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.371648 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.371660 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.371672 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.372022 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.372358 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.372362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.372365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.372612 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.373110 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.373117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.373529 (+ 412us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.373764 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.374192 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":50,"spinlock_wait_cycles":3968,"thread_start_us":112,"threads_started":1}
I20260929 03:10:46.376842 28011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.124@KRBTEST.COM for oryx/127.20.174.125@KRBTEST.COM
I20260929 03:10:46.386612 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.377405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:43575 (local address 127.0.0.1:33834)
0929 03:10:46.377760 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.377776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.377850 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.378101 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.378103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.378253 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.378357 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.378362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.379421 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.379425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.380030 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.380040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.380526 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.380529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.380609 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.381176 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.381188 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.382725 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.384996 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.384999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.385001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.385217 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.385532 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.385534 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.385536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.385613 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.386099 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.386103 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.386202 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.386292 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.386419 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260929 03:10:46.386674 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.377514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.20.174.125:43575)
0929 03:10:46.377791 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.377796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.377813 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:10:46.377946 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.377949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.377995 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.378073 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.378549 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.379254 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.380154 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.380421 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.380506 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.382972 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.382976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.382978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.383007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.384868 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.385346 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.385349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.385350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.385441 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.385692 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.385695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.385696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.385944 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.386105 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.386355 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.386521 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
I20260929 03:10:46.387544 28011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.124@KRBTEST.COM for oryx/127.20.174.124@KRBTEST.COM
I20260929 03:10:46.395465 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.388130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:34329 (local address 127.0.0.1:40190)
0929 03:10:46.388263 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.388279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.388350 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.388530 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.388532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.388663 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.388746 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.388751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.389498 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.389501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.390141 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.390150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.390732 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.390735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.390800 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.391299 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.391310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.392592 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.394135 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.394138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.394139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.394323 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.394644 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.394646 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.394647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.394721 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.395053 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.395055 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.395129 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.395207 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.395308 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 03:10:46.395509 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.388216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.20.174.124:34329)
0929 03:10:46.388337 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.388341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.388353 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:10:46.388418 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.388421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.388457 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.388521 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.388833 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.389370 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.390259 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.390613 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.390695 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.392736 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.392739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.392741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.392763 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.394034 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.394405 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.394407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.394409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.394562 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.394798 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.394800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.394801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.394978 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.395055 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.395261 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.395356 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:10:46.399021 28011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: Bootstrap starting.
I20260929 03:10:46.401451 28011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:46.402283 28011 log.cc:824] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:46.404284 28011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: No bootstrap required, opened a new log
I20260929 03:10:46.407387 28011 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } }
I20260929 03:10:46.407617 28011 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:46.407694 28011 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2be03d77f4340b79c5056a541959897, State: Initialized, Role: FOLLOWER
I20260929 03:10:46.408149 28011 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } }
I20260929 03:10:46.408680 28017 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } } }
I20260929 03:10:46.408802 28017 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:10:46.409420 28011 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:10:46.414610 28028 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:10:46.414702 28028 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:10:46.418408 27859 master_runner.cc:428] Detected that this master d2be03d77f4340b79c5056a541959897 is joining an existing cluster
I20260929 03:10:46.418509 27859 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.174.124:34329
I20260929 03:10:46.429102 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 27347
I20260929 03:10:46.431854 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.421839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.20.174.126:44627)
0929 03:10:46.421962 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.421966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.422200 (+ 234us) server_negotiation.cc:407] Connection header received
0929 03:10:46.422359 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.422363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.422416 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.422534 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.423244 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.424113 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.425401 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.425773 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.425858 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.427500 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.427503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.427505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.427530 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.429510 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.430640 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.430647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.430652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.430747 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.431007 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.431012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.431014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.431205 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.431313 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.431582 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.431699 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:10:46.431869 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.421227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:43824)
0929 03:10:46.422104 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.422120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.422238 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.422697 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.422700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.422945 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.423095 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.423103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.424261 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.424264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.425243 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.425255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.426038 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.426042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.426139 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.426817 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.426837 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.427349 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.430216 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.430222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.430235 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.430516 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.430841 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.430843 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.430844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.430914 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.431313 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.431316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.431416 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.431522 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.431674 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":124,"threads_started":1}
I20260929 03:10:46.436657 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:44577
--local_ip_for_outbound_sockets=127.20.174.65
--tserver_master_addrs=127.20.174.126:44627
--webserver_port=41333
--webserver_interface=127.20.174.65
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab
--principal=oryx/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.174.126:44627,127.20.174.125:43575
--tserver_master_addrs=127.20.174.126:44627,127.20.174.125:43575,127.20.174.124:34329 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651444, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.126@KRBTEST.COM for oryx/127.20.174.124@KRBTEST.COM
I20260929 03:10:46.452744 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.422070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:43575 (local address 127.0.0.1:33850)
0929 03:10:46.422651 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.422666 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.422794 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.423192 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.423194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.423399 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.423503 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.423509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.424547 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.424550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.425513 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.425522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.426231 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.426235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.426375 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.447251 (+ 20876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.447270 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.448061 (+ 791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.450319 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.450322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.450325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.450907 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.451450 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.451453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.451455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.451540 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.452093 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.452096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.452199 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.452295 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.452474 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":57,"spinlock_wait_cycles":3328,"thread_start_us":107,"threads_started":1}
I20260929 03:10:46.453694 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.421301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.20.174.125:43575)
0929 03:10:46.421473 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.421478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.422738 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 03:10:46.422899 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.422903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.422969 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.423027 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.423624 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.424403 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.425647 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.426017 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.426094 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.448269 (+ 22175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.448273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.448275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.448309 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.450167 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.451211 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.451215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.451217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.451317 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.451652 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.451656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.451658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.451963 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.452050 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.453369 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.453542 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:10:46.454308 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.435778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:34329 (local address 127.0.0.1:40218)
0929 03:10:46.435976 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.435993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.436087 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.438636 (+ 2549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.438641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.438870 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.439010 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.439018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.440093 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.440096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.440702 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.440711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.441206 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.441210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.441312 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.441828 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.441841 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.443185 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.452017 (+ 8832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.452023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.452025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.452309 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.452713 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.452717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.452719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.452857 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.453546 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.453549 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.453704 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.453859 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.454110 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 03:10:46.455260 28037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.435865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.20.174.124:34329)
0929 03:10:46.438256 (+ 2391us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.438264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.438279 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:46.438374 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.438378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.438458 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.438555 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.439148 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.439974 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.440829 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.441087 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.441200 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.443359 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.443363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.443365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.443387 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.451862 (+ 8475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.452425 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.452431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.452434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.452509 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.453086 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.453089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.453091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.453370 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.454745 (+ 1375us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.454843 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.455011 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2305,"thread_start_us":98,"threads_started":1}
I20260929 03:10:46.456480 27285 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.174.124:34329: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d2be03d77f4340b79c5056a541959897"
member_type: NON_VOTER
last_known_addr {
host: "127.20.174.124"
port: 34329
}
attrs {
promote: true
}
}
dest_uuid: "51489f943cd34cbb87a0d04a58ab918c"
cas_config_opid_index: 5
I20260929 03:10:46.456787 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.421440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.20.174.124:34329)
0929 03:10:46.421544 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.421549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.422550 (+ 1001us) server_negotiation.cc:407] Connection header received
0929 03:10:46.422696 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.422701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.422740 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.422789 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.423413 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.424143 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.425745 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.426035 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.426097 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.448232 (+ 22135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.448236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.448238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.448261 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.451083 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.455656 (+ 4573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.455659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.455661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.455744 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.456009 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.456012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.456013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.456200 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.456274 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.456563 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.456666 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:10:46.457195 27285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } }
I20260929 03:10:46.457607 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.421627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:34329 (local address 127.0.0.1:40202)
0929 03:10:46.422472 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.422487 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.422599 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.422919 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.422921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.423159 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.423297 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.423305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.424266 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.424268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.425647 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.425654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.426240 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.426243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.426401 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.447145 (+ 20744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.447165 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.448061 (+ 896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.455087 (+ 7026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.455089 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.455092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.455551 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.455840 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.455843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.455844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.455923 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.456314 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.456316 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.456395 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.457227 (+ 832us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.457442 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":75,"spinlock_wait_cycles":8448,"thread_start_us":424,"threads_started":1}
I20260929 03:10:46.458514 27845 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d2be03d77f4340b79c5056a541959897 (127.20.174.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.459903 27845 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.174.124:34329
I20260929 03:10:46.460124 27845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.460238 27845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:46.460768 27859 master.cc:537] Master@127.20.174.124:34329 shutting down...
W20260929 03:10:46.461990 27270 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 03:10:46.465548 27859 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:46.466015 27859 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:46.466219 27859 tablet_replica.cc:330] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: stopping tablet replica
I20260929 03:10:46.467402 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.458514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:43575 (local address 127.0.0.1:33860)
0929 03:10:46.461949 (+ 3435us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.461966 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.462047 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.462526 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.462529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.462546 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:46.462643 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.462649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.464416 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.464422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.465657 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.465668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.466939 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.466943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.467056 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.467112 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.467185 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.467272 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3338}
I20260929 03:10:46.467926 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.460409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.20.174.125:43575)
0929 03:10:46.462219 (+ 1810us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.462226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.462243 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:10:46.462294 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.462297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.462365 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.462503 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:46.462904 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.463991 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.465814 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.466494 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.466612 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.467670 (+ 1058us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.467728 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.467806 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1720}
I20260929 03:10:46.468461 27903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 LEARNER]: Refusing update from remote peer 51489f943cd34cbb87a0d04a58ab918c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 03:10:46.469156 27847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [LEADER]: Connected to new peer: Peer: permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 03:10:46.470458 27903 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d2be03d77f4340b79c5056a541959897 (127.20.174.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.472146 28041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.472339 28041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:10:46.472627 27847 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c: attempting to promote NON_VOTER 573016e12e7e4aa9b3e8161371367e69 to VOTER
I20260929 03:10:46.472616 27932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51489f943cd34cbb87a0d04a58ab918c. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: NON_VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: true } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.472718 27932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:10:46.473153 27847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } }
I20260929 03:10:46.473685 27859 master.cc:552] Master@127.20.174.124:34329 shutdown complete.
I20260929 03:10:46.473735 27859 master_runner.cc:305] Clearing existing system tablet
I20260929 03:10:46.473973 27859 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:46.474026 27859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:46.474526 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.474147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:34329 (local address 127.0.0.1:40230)
0929 03:10:46.474316 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.474417 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:34329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 03:10:46.474660 27903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 LEARNER]: Refusing update from remote peer 51489f943cd34cbb87a0d04a58ab918c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 03:10:46.474989 27846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [LEADER]: Connected to new peer: Peer: permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 03:10:46.476516 27847 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 573016e12e7e4aa9b3e8161371367e69 (127.20.174.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.477797 27847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.477903 27847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:46.478030 27845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.478121 27845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:46.478673 27903 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 573016e12e7e4aa9b3e8161371367e69 (127.20.174.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.480098 27932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.480222 27932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:10:46.483526 27859 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260929 03:10:46.484025 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.483646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:34329 (local address 127.0.0.1:40246)
0929 03:10:46.483881 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.483940 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:34329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260929 03:10:46.484189 27270 consensus_peers.cc:487] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c -> Peer d2be03d77f4340b79c5056a541959897 (127.20.174.124:34329): Couldn't send request to peer d2be03d77f4340b79c5056a541959897. Status: Network error: Client connection negotiation failed: client connection to 127.20.174.124:34329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 03:10:46.484794 28043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:46.484994 27859 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:46.485119 27859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:10:46.485155 27859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/wal
uuid: "d2be03d77f4340b79c5056a541959897"
format_stamp: "Formatted at 2026-09-29 03:10:46 on dist-test-slave-k1dg"
server_key: "c44cf2d6c01a7b209ba18b533c203a82"
server_key_iv: "b677d7f69c258e4e8601e727659d8566"
server_key_version: "encryptionkey@0"
I20260929 03:10:46.487120 27859 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:46.489156 27859 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:10:46.489269 27859 log.cc:1196] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 03:10:46.489497 27859 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: Deleting consensus metadata
I20260929 03:10:46.489666 27859 master_runner.cc:315] Copying system tablet from 127.20.174.126:44627
I20260929 03:10:46.490863 27859 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: tablet copy: Beginning tablet copy session from remote peer at address 127.20.174.126:44627
I20260929 03:10:46.498870 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.491461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.20.174.126:44627)
0929 03:10:46.491587 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.491592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.491806 (+ 214us) server_negotiation.cc:407] Connection header received
0929 03:10:46.491956 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.491959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.492004 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.492095 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.492553 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.493174 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.494021 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.494257 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.494366 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.495564 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.495568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.495570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.495596 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.497049 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.497477 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.497482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.497484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.497562 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.497846 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.497849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.497851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.498139 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.498231 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.498585 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.498702 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:10:46.498880 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.491420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:43832)
0929 03:10:46.491718 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.491730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.491854 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.492126 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.492129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.492324 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.492436 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.492442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.493285 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.493290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.493886 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.493894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.494368 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.494372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.494514 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.495077 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.495089 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.495423 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.497167 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.497171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.497174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.497377 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.497659 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.497663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.497665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.497753 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.498221 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.498224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.498336 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.498512 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.498674 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":84,"threads_started":1}
I20260929 03:10:46.499768 27305 tablet_copy_service.cc:140] P 51489f943cd34cbb87a0d04a58ab918c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d2be03d77f4340b79c5056a541959897 ({username='oryx', principal='oryx/127.20.174.124@KRBTEST.COM'} at 127.0.0.1:43832)
I20260929 03:10:46.499886 27305 tablet_copy_service.cc:161] P 51489f943cd34cbb87a0d04a58ab918c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d2be03d77f4340b79c5056a541959897 at {username='oryx', principal='oryx/127.20.174.124@KRBTEST.COM'} at 127.0.0.1:43832: session id = d2be03d77f4340b79c5056a541959897-00000000000000000000000000000000
I20260929 03:10:46.500735 27305 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:10:46.502061 27859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:10:46.504280 27859 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: tablet copy: Starting download of 0 data blocks...
I20260929 03:10:46.504523 27859 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: tablet copy: Starting download of 1 WAL segments...
I20260929 03:10:46.506009 27859 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:10:46.508298 27305 tablet_copy_service.cc:342] P 51489f943cd34cbb87a0d04a58ab918c: Request end of tablet copy session d2be03d77f4340b79c5056a541959897-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.174.124@KRBTEST.COM'} at 127.0.0.1:43832
I20260929 03:10:46.508431 27305 tablet_copy_service.cc:433] P 51489f943cd34cbb87a0d04a58ab918c: ending tablet copy session d2be03d77f4340b79c5056a541959897-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d2be03d77f4340b79c5056a541959897
I20260929 03:10:46.516368 27859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:46.518539 28050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:46.518551 28051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:46.518929 27859 server_base.cc:1034] running on GCE node
W20260929 03:10:46.519165 28053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:46.519549 27859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:46.520032 27859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:46.521221 27859 hybrid_clock.cc:648] HybridClock initialized: now 1790651446521209 us; error 29 us; skew 500 ppm
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:46.522970 27859 init.cc:377] Logged in from keytab as oryx/127.20.174.124@KRBTEST.COM (short username oryx)
I20260929 03:10:46.523927 27859 webserver.cc:460] Webserver started at http://127.20.174.124:36853/ using document root <none> and password file <none>
I20260929 03:10:46.524147 27859 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:46.524200 27859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:46.525444 27859 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:46.526423 28060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:46.526688 27859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:46.526757 27859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/wal
uuid: "d2be03d77f4340b79c5056a541959897"
format_stamp: "Formatted at 2026-09-29 03:10:46 on dist-test-slave-k1dg"
server_key: "c44cf2d6c01a7b209ba18b533c203a82"
server_key_iv: "b677d7f69c258e4e8601e727659d8566"
server_key_version: "encryptionkey@0"
I20260929 03:10:46.526844 27859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:46.535528 27859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:46.537643 27859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:46.542665 27859 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.124:34329
I20260929 03:10:46.543246 27859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
I20260929 03:10:46.546689 28112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.124:34329 every 8 connection(s)
I20260929 03:10:46.548178 28113 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:10:46.548987 27610 heartbeater.cc:503] Master 127.20.174.126:44627 was elected leader, sending a full tablet report...
I20260929 03:10:46.549170 28113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: Bootstrap starting.
I20260929 03:10:46.558619 28113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:10:46.558934 28113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: Bootstrap complete.
I20260929 03:10:46.559427 28113 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } }
I20260929 03:10:46.559564 28113 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d2be03d77f4340b79c5056a541959897, State: Initialized, Role: LEARNER
I20260929 03:10:46.559674 28113 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } }
I20260929 03:10:46.560032 28116 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.560199 28113 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:46.560207 28116 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:10:46.564045 28127 catalog_manager.cc:1259] Loaded cluster ID: f9cd3cde7b3b45f2a0a453c7770a66f0
I20260929 03:10:46.564133 28127 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: loading cluster ID for follower catalog manager: success
I20260929 03:10:46.565726 28127 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: acquiring CA information for follower catalog manager: success
I20260929 03:10:46.566911 28127 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:10:46.619109 28038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:46.619515 28038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:46.619608 28038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:46.624980 28038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:46.625048 28038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:46.625154 28038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:46.631098 28038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab
--principal=oryx/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:44577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=41333
--tserver_master_addrs=127.20.174.126:44627,127.20.174.125:43575,127.20.174.124:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:46.632228 28038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:46.633308 28038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:46.643191 28133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:46.643196 28135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:46.643196 28132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:46.643595 28038 server_base.cc:1034] running on GCE node
I20260929 03:10:46.644049 28038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:46.644805 28038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:46.645991 28038 hybrid_clock.cc:648] HybridClock initialized: now 1790651446645977 us; error 32 us; skew 500 ppm
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:46.648923 28038 init.cc:377] Logged in from keytab as oryx/127.20.174.65@KRBTEST.COM (short username oryx)
I20260929 03:10:46.649792 28038 webserver.cc:460] Webserver started at http://127.20.174.65:41333/ using document root <none> and password file <none>
I20260929 03:10:46.650326 28038 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:46.650398 28038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:46.654402 28038 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:46.656687 28142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:46.657920 28038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 03:10:46.658042 28038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "23541f7f22ce42debe52d0985731995b"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "a6c0aec3b74042b6e4860f041fecc011"
server_key_iv: "5faf04f0bf089c1214ce1f931fb22a39"
server_key_version: "encryptionkey@0"
I20260929 03:10:46.658646 28038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:46.678602 28038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:46.682358 28038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:46.682647 28038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:46.683350 28038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:46.684438 28038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:46.684490 28038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:46.684563 28038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:46.684597 28038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:46.694258 28038 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:44577
I20260929 03:10:46.694288 28255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:44577 every 8 connection(s)
I20260929 03:10:46.695323 28038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:46.696257 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 28038
I20260929 03:10:46.696478 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 27481
I20260929 03:10:46.706137 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:35397
--local_ip_for_outbound_sockets=127.20.174.66
--tserver_master_addrs=127.20.174.126:44627
--webserver_port=37027
--webserver_interface=127.20.174.66
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab
--principal=oryx/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.174.126:44627,127.20.174.125:43575
--tserver_master_addrs=127.20.174.126:44627,127.20.174.125:43575,127.20.174.124:34329 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.124@KRBTEST.COM
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.125@KRBTEST.COM
I20260929 03:10:46.715392 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.700064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:51981 (local address 127.20.174.124:34329)
0929 03:10:46.700665 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.700670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.700916 (+ 246us) server_negotiation.cc:407] Connection header received
0929 03:10:46.701104 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.701110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.701166 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.701232 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.702667 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.703730 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.705751 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.706130 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.706197 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.710327 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.710333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.710338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.710377 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.712333 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.713295 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.713300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.713303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.713401 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.713828 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.713834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.713838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.714105 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.714187 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.715027 (+ 840us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.715180 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":73,"threads_started":1}
I20260929 03:10:46.715796 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.697202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:39525 (local address 127.20.174.126:44627)
0929 03:10:46.697356 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.697363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.699243 (+ 1880us) server_negotiation.cc:407] Connection header received
0929 03:10:46.700436 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.700443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.700516 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.700612 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.702669 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.703625 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.705774 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.706122 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.706205 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.711903 (+ 5698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.711908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.711912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.711949 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.713770 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.714229 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.714232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.714234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.714325 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.714840 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.714844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.714845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.715085 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.715168 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.715523 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.715643 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:10:46.716503 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.697993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:51539 (local address 127.20.174.125:43575)
0929 03:10:46.698931 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.698938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.699294 (+ 356us) server_negotiation.cc:407] Connection header received
0929 03:10:46.700395 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.700399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.700471 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.700530 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.702668 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.703502 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.705752 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.706105 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.706188 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.710422 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.710453 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.710460 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.710498 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.712537 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.713234 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.713240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.713244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.713348 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.713835 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.713840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.713844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.714126 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.714238 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.716242 (+ 2004us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.716370 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831}
I20260929 03:10:46.716790 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.699935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:34329 (local address 127.20.174.65:51981)
0929 03:10:46.700837 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.700851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.700986 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.701388 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.701391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.701634 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.702491 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.702513 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.703855 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.703858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.705599 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.705614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.706400 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.706404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.706672 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.707416 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.707435 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.710106 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.712499 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.712506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.712525 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.713143 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.713526 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.713529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.713532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.713691 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.714286 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.714294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.714731 (+ 437us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.715813 (+ 1082us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.716202 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":564,"spinlock_wait_cycles":7168,"thread_start_us":111,"threads_started":1}
I20260929 03:10:46.717036 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.697785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.65:39525)
0929 03:10:46.698969 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.699013 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.700247 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.700879 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.700891 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.701631 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.702489 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.702514 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.703780 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.703786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.705621 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.705636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.706751 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.706757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.706893 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.707596 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.707616 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.710664 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.713890 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.713893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.713895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.714115 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.714618 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.714621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.714623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.714709 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.715224 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.715227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.715309 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.716006 (+ 697us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.716842 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":1110,"spinlock_wait_cycles":17024,"thread_start_us":139,"threads_started":1}
I20260929 03:10:46.717118 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.697785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:43575 (local address 127.20.174.65:51539)
0929 03:10:46.698953 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.699013 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.700138 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.700879 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.700891 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.701601 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.702495 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.702513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.703674 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.703677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.705623 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.705629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.706400 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.706404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.706792 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.707403 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.707426 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.710260 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.712687 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.712693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.712699 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.713092 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.713470 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.713475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.713477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.713707 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.714288 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.714301 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.714734 (+ 433us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.714995 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.715446 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":509,"spinlock_wait_cycles":39552,"thread_start_us":123,"threads_started":1}
I20260929 03:10:46.718580 28258 heartbeater.cc:347] Connected to a master server at 127.20.174.125:43575
I20260929 03:10:46.718560 28257 heartbeater.cc:347] Connected to a master server at 127.20.174.124:34329
I20260929 03:10:46.718830 28257 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:46.718555 28256 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44627
I20260929 03:10:46.718895 28256 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:46.719376 28256 heartbeater.cc:511] Master 127.20.174.126:44627 requested a full tablet report, sending...
I20260929 03:10:46.718832 28258 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:46.720201 28258 heartbeater.cc:511] Master 127.20.174.125:43575 requested a full tablet report, sending...
I20260929 03:10:46.720397 28257 heartbeater.cc:511] Master 127.20.174.124:34329 requested a full tablet report, sending...
I20260929 03:10:46.720883 27285 ts_manager.cc:194] Re-registered known tserver with Master: 23541f7f22ce42debe52d0985731995b (127.20.174.65:44577)
I20260929 03:10:46.721478 28077 ts_manager.cc:194] Registered new tserver with Master: 23541f7f22ce42debe52d0985731995b (127.20.174.65:44577)
I20260929 03:10:46.721488 27893 ts_manager.cc:194] Registered new tserver with Master: 23541f7f22ce42debe52d0985731995b (127.20.174.65:44577)
I20260929 03:10:46.721660 27285 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:39525
I20260929 03:10:46.792724 27744 heartbeater.cc:503] Master 127.20.174.126:44627 was elected leader, sending a full tablet report...
W20260929 03:10:46.828073 28264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:46.828339 28264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:46.828413 28264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:46.832538 28264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:46.832604 28264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:46.832707 28264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:46.838164 28264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab
--principal=oryx/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:35397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=37027
--tserver_master_addrs=127.20.174.126:44627,127.20.174.125:43575,127.20.174.124:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:46.839283 28264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:46.840324 28264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:46.850224 28269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:46.850558 28272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:46.850589 28270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:46.850864 28264 server_base.cc:1034] running on GCE node
I20260929 03:10:46.851291 28264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:46.851980 28264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:46.853147 28264 hybrid_clock.cc:648] HybridClock initialized: now 1790651446853116 us; error 54 us; skew 500 ppm
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:46.856055 28264 init.cc:377] Logged in from keytab as oryx/127.20.174.66@KRBTEST.COM (short username oryx)
I20260929 03:10:46.857009 28264 webserver.cc:460] Webserver started at http://127.20.174.66:37027/ using document root <none> and password file <none>
I20260929 03:10:46.857614 28264 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:46.857722 28264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:46.862056 28264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:46.864368 28279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:46.865805 28264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:46.865967 28264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "b0c90b2b03d847c8a4a68151a867bb2e"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "6a69e08cfa68a0a67a6b18dbd959eece"
server_key_iv: "75f68f710fcf3a2c62073ca13933b69c"
server_key_version: "encryptionkey@0"
I20260929 03:10:46.866631 28264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:46.883440 28264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:46.887024 28264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:46.887233 28264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:46.887898 28264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:46.888814 28264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:46.888859 28264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:46.888926 28264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:46.888957 28264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:46.898065 28264 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:35397
I20260929 03:10:46.898087 28392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:35397 every 8 connection(s)
I20260929 03:10:46.899183 28264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:46.904496 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 28264
I20260929 03:10:46.904675 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 27615
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.124@KRBTEST.COM
I20260929 03:10:46.912792 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:35159
--local_ip_for_outbound_sockets=127.20.174.67
--tserver_master_addrs=127.20.174.126:44627
--webserver_port=43513
--webserver_interface=127.20.174.67
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab
--principal=oryx/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.174.126:44627,127.20.174.125:43575
--tserver_master_addrs=127.20.174.126:44627,127.20.174.125:43575,127.20.174.124:34329 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
Sep 29 03:10:46 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651446, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.125@KRBTEST.COM
I20260929 03:10:46.918350 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.901009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:55415 (local address 127.20.174.125:43575)
0929 03:10:46.901142 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.901147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.902275 (+ 1128us) server_negotiation.cc:407] Connection header received
0929 03:10:46.903485 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.903488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.903543 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.903597 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.906208 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.907061 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.909224 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.909716 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.909807 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.914627 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.914633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.914637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.914673 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.916502 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.916991 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.916998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.917000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.917083 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.917379 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.917382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.917384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.917588 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.917655 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.918051 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.918179 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:10:46.918350 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.901366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:55129 (local address 127.20.174.126:44627)
0929 03:10:46.901629 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.901633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.902504 (+ 871us) server_negotiation.cc:407] Connection header received
0929 03:10:46.903470 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.903474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.903530 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.903632 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.906568 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.907313 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.909224 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.909716 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.909798 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.914058 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.914063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.914069 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.914108 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.916084 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.916619 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.916622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.916625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.916706 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.917014 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.917017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.917019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.917252 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.917340 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.918055 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.918179 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260929 03:10:46.918481 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.901495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:56677 (local address 127.20.174.124:34329)
0929 03:10:46.901680 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.901684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.902772 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 03:10:46.903485 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.903488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.903544 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.903597 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:46.906042 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.907013 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.909227 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.909813 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.909911 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.913489 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:46.913492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:46.913496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:46.913531 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:46.915857 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.916482 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.916488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.916493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.916585 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:46.917009 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:46.917016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:46.917020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:46.917286 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:46.917364 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.918063 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.918220 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 03:10:46.920325 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.901528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:34329 (local address 127.20.174.66:56677)
0929 03:10:46.902675 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.902686 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.903266 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.903805 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.903812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.904377 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.905847 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.905871 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.907209 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.907215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.909073 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.909087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.910103 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.910109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.910321 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.911145 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.911180 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.913310 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.916005 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.916013 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.916031 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.916373 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.916689 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.916695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.916699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.916897 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.917441 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.917446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.917770 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.918651 (+ 881us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.919076 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":1355,"spinlock_wait_cycles":68224,"thread_start_us":103,"threads_started":1}
I20260929 03:10:46.920342 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.901528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:43575 (local address 127.20.174.66:55415)
0929 03:10:46.902119 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.902171 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.903268 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.903805 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.903812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.904377 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.906071 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.906083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.907209 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.907212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.909077 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.909088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.910013 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.910016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.910308 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.911304 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.911319 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.914421 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.916615 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.916618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.916620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.916861 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.917195 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.917198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.917200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.917276 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.917726 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.917729 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.917795 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.918625 (+ 830us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.919310 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":1811,"spinlock_wait_cycles":11648,"thread_start_us":179,"threads_started":1}
I20260929 03:10:46.920342 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.901528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.66:55129)
0929 03:10:46.902404 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.902415 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.903262 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.903803 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.903811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.904361 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:46.905847 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.905871 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.907457 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.907460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.909077 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.909088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.910013 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.910016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.910272 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.911186 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:46.911202 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:46.913879 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:46.916208 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.916211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.916214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.916495 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.916820 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:46.916823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:46.916826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:46.916912 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:46.917401 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:46.917410 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:46.917759 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.918625 (+ 866us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.919211 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":767,"spinlock_wait_cycles":73344,"thread_start_us":184,"threads_started":1}
I20260929 03:10:46.921717 28395 heartbeater.cc:347] Connected to a master server at 127.20.174.125:43575
I20260929 03:10:46.921762 28394 heartbeater.cc:347] Connected to a master server at 127.20.174.124:34329
I20260929 03:10:46.921767 28393 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44627
I20260929 03:10:46.921960 28395 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:46.921960 28393 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:46.921959 28394 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:46.922638 28395 heartbeater.cc:511] Master 127.20.174.125:43575 requested a full tablet report, sending...
I20260929 03:10:46.922631 28394 heartbeater.cc:511] Master 127.20.174.124:34329 requested a full tablet report, sending...
I20260929 03:10:46.922629 28393 heartbeater.cc:511] Master 127.20.174.126:44627 requested a full tablet report, sending...
I20260929 03:10:46.923920 28077 ts_manager.cc:194] Registered new tserver with Master: b0c90b2b03d847c8a4a68151a867bb2e (127.20.174.66:35397)
I20260929 03:10:46.924088 27893 ts_manager.cc:194] Registered new tserver with Master: b0c90b2b03d847c8a4a68151a867bb2e (127.20.174.66:35397)
I20260929 03:10:46.924530 27285 ts_manager.cc:194] Re-registered known tserver with Master: b0c90b2b03d847c8a4a68151a867bb2e (127.20.174.66:35397)
I20260929 03:10:46.925274 27285 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:55129
I20260929 03:10:46.973155 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.969117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.20.174.124:34329)
0929 03:10:46.969257 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:46.969262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:46.969280 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:10:46.969481 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:46.969484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:46.969543 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:46.969652 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:46.969955 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.970607 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.971758 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.972367 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.972551 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.972752 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:46.972934 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 03:10:46.973005 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:10:46.973166 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:46.969030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:34329 (local address 127.0.0.1:40248)
0929 03:10:46.969169 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:46.969186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:46.969316 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:46.969691 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:46.969694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:46.969717 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:46.969829 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.969835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.970729 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.970733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.971617 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:46.971629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:46.972595 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:46.972599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:46.972764 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:46.972807 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:10:46.972918 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 03:10:46.973015 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 03:10:46.974220 28087 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 03:10:46.974717 28116 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51489f943cd34cbb87a0d04a58ab918c. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:46.974877 28116 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: This master's current role is: LEARNER
W20260929 03:10:47.037901 28400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:47.038213 28400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:47.038285 28400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:47.043224 28400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:47.043326 28400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:47.043426 28400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:47.049170 28400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab
--principal=oryx/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:35159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=43513
--tserver_master_addrs=127.20.174.126:44627,127.20.174.125:43575,127.20.174.124:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:47.050395 28400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:47.051515 28400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:47.061909 28405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:47.061901 28408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:47.062325 28400 server_base.cc:1034] running on GCE node
W20260929 03:10:47.061901 28406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:47.062861 28400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:47.063601 28400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:47.064785 28400 hybrid_clock.cc:648] HybridClock initialized: now 1790651447064778 us; error 70 us; skew 500 ppm
Sep 29 03:10:47 dist-test-slave-k1dg krb5kdc[27240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651447, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:47.067759 28400 init.cc:377] Logged in from keytab as oryx/127.20.174.67@KRBTEST.COM (short username oryx)
I20260929 03:10:47.068626 28400 webserver.cc:460] Webserver started at http://127.20.174.67:43513/ using document root <none> and password file <none>
I20260929 03:10:47.069164 28400 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:47.069267 28400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:47.073482 28400 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:47.075663 28415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:47.077114 28400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:47.077226 28400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "1aad6b620fd141db897db0570f975caf"
format_stamp: "Formatted at 2026-09-29 03:10:45 on dist-test-slave-k1dg"
server_key: "8c4fd9e207746c101125bf8e8fa08d09"
server_key_iv: "ee97e29348cd6d97a3ff6f35f4e0f39f"
server_key_version: "encryptionkey@0"
I20260929 03:10:47.077792 28400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:47.094746 28400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:47.097863 28400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:47.098024 28400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:47.098642 28400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:47.099527 28400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:47.099578 28400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:47.099620 28400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:47.099637 28400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:47.109278 28400 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:35159
I20260929 03:10:47.109295 28528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:35159 every 8 connection(s)
I20260929 03:10:47.110515 28400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:47.111672 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 28400
Sep 29 03:10:47 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651447, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.125@KRBTEST.COM
Sep 29 03:10:47 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651447, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:47.127624 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.114490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.20.174.124:34329)
0929 03:10:47.114897 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.114902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.115145 (+ 243us) server_negotiation.cc:407] Connection header received
0929 03:10:47.115379 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.115382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.115438 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.115500 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:47.116679 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.117515 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.118871 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.119205 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.119284 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.122373 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:47.122377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:47.122380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:47.122409 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:47.124509 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.125494 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.125499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.125503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.125602 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.125965 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.125970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.125973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.126210 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:47.126326 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.127200 (+ 874us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.127368 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":82,"threads_started":1}
Sep 29 03:10:47 dist-test-slave-k1dg krb5kdc[27240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651447, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.124@KRBTEST.COM
I20260929 03:10:47.130018 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.112782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:39225 (local address 127.20.174.125:43575)
0929 03:10:47.113053 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.113057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.114376 (+ 1319us) server_negotiation.cc:407] Connection header received
0929 03:10:47.115815 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.115818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.115875 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.115947 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:47.117749 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.118499 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.120399 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.120699 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.120809 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.124180 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:47.124183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:47.124186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:47.124211 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:47.126818 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.127508 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.127515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.127520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.127609 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.128076 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.128080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.128082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.128315 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:47.128424 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.129702 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.129847 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260929 03:10:47.130714 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.113070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:43575 (local address 127.20.174.67:39225)
0929 03:10:47.114196 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:47.114253 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:47.115468 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:47.116279 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:47.116289 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:47.116878 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:47.117577 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.117598 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.118667 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.118671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.120274 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.120287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.121018 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.121022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.121339 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.122013 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:47.122034 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:47.124014 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:47.126966 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:47.126973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:47.126990 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:47.127379 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:47.127736 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:47.127742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:47.127744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:47.127948 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:47.128487 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:47.128495 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:47.128859 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 03:10:47.129104 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 03:10:47.129488 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":130,"spinlock_wait_cycles":6912,"thread_start_us":130,"threads_started":1}
I20260929 03:10:47.132021 28531 heartbeater.cc:347] Connected to a master server at 127.20.174.125:43575
I20260929 03:10:47.132268 28531 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:47.132777 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.114097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:33797 (local address 127.20.174.126:44627)
0929 03:10:47.114221 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.114225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.114363 (+ 138us) server_negotiation.cc:407] Connection header received
0929 03:10:47.116913 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.116917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.116976 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.117031 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:47.119561 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.120691 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.124628 (+ 3937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.124939 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.125045 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.127783 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:47.127790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:47.127794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:47.127819 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:47.129621 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.131212 (+ 1591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.131218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.131220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.131309 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.131636 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.131642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.131645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.131938 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:47.132033 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.132514 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.132636 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:10:47.132802 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.113069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.67:33797)
0929 03:10:47.114193 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:47.114255 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:47.116450 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:47.117172 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:47.117174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:47.117357 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:47.117578 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.117600 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.121147 (+ 3547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.121150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.124119 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.124131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.125181 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.125187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.125304 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.125778 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:47.125798 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:47.127596 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:47.130493 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:47.130499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:47.130504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:47.131081 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:47.131430 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:47.131433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:47.131437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:47.131527 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:47.132102 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:47.132105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:47.132199 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 03:10:47.132400 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 03:10:47.132566 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":3,"thread_start_us":142,"threads_started":1}
I20260929 03:10:47.132968 28531 heartbeater.cc:511] Master 127.20.174.125:43575 requested a full tablet report, sending...
I20260929 03:10:47.133984 28529 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44627
I20260929 03:10:47.134078 28529 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:47.134188 27893 ts_manager.cc:194] Registered new tserver with Master: 1aad6b620fd141db897db0570f975caf (127.20.174.67:35159)
I20260929 03:10:47.134263 28529 heartbeater.cc:511] Master 127.20.174.126:44627 requested a full tablet report, sending...
I20260929 03:10:47.134969 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.112917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:55301 (local address 127.20.174.124:34329)
0929 03:10:47.113153 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.113157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.115885 (+ 2728us) server_negotiation.cc:407] Connection header received
0929 03:10:47.116069 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.116072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.116127 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.116186 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:47.117967 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.118742 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.124344 (+ 5602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.124699 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.124775 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.129889 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:47.129893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:47.129896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:47.129920 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:47.131685 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.133528 (+ 1843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.133533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.133537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.133626 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.133950 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.133953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.133955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.134191 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:47.134283 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.134677 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.134826 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260929 03:10:47.134956 27285 ts_manager.cc:194] Re-registered known tserver with Master: 1aad6b620fd141db897db0570f975caf (127.20.174.67:35159)
I20260929 03:10:47.135408 28541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.114626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.20.174.125:43575)
0929 03:10:47.116021 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.116026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.116037 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:10:47.116066 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.116069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.116100 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.116171 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:47.117158 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.117948 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.119866 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.120227 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.120337 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.122996 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:47.123001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:47.123004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:47.123032 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:47.132617 (+ 9585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.133259 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.133263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.133265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.133360 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.133679 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.133682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.133684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.133901 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:47.134026 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.134886 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.135269 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1331,"thread_start_us":59,"threads_started":1}
I20260929 03:10:47.135450 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.113226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:34329 (local address 127.20.174.67:55301)
0929 03:10:47.115801 (+ 2575us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:47.115815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:47.115953 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:47.116361 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:47.116364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:47.117721 (+ 1357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:47.117841 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.117847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.118893 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.118896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.122546 (+ 3650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.122557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.124923 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.124926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.125040 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.126723 (+ 1683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:47.126738 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:47.129717 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:47.133088 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:47.133092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:47.133094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:47.133404 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:47.133739 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:47.133742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:47.133744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:47.133841 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:47.134355 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:47.134358 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:47.134466 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 03:10:47.135122 (+ 656us) client_negotiation.cc:240] Negotiation successful
0929 03:10:47.135273 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2498,"mutex_wait_us":2475,"thread_start_us":652,"threads_started":1}
I20260929 03:10:47.135677 27285 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:33797
I20260929 03:10:47.136186 28530 heartbeater.cc:347] Connected to a master server at 127.20.174.124:34329
I20260929 03:10:47.136282 28530 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:47.136448 28530 heartbeater.cc:511] Master 127.20.174.124:34329 requested a full tablet report, sending...
I20260929 03:10:47.137449 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.115037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.20.174.126:44627)
0929 03:10:47.115572 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.115576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.118864 (+ 3288us) server_negotiation.cc:407] Connection header received
0929 03:10:47.118911 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.118914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.118976 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.119061 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:47.122013 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.122843 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.124009 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.124360 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.124474 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.126526 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:47.126529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:47.126532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:47.126559 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:47.130844 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.131709 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.131712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.131714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.131821 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.136181 (+ 4360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.136186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.136188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.136416 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:47.136549 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.137161 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.137287 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":77,"threads_started":1}
I20260929 03:10:47.138134 28077 ts_manager.cc:194] Registered new tserver with Master: 1aad6b620fd141db897db0570f975caf (127.20.174.67:35159)
I20260929 03:10:47.140726 27285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:47.143890 27285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:47.146555 27847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: NON_VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: true } } }
I20260929 03:10:47.146701 27847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:47.150532 27847 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c: attempting to promote NON_VOTER d2be03d77f4340b79c5056a541959897 to VOTER
I20260929 03:10:47.150911 27847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } }
I20260929 03:10:47.152186 27903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 FOLLOWER]: Refusing update from remote peer 51489f943cd34cbb87a0d04a58ab918c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 03:10:47.152645 28086 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 1 LEARNER]: Refusing update from remote peer 51489f943cd34cbb87a0d04a58ab918c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 03:10:47.152606 28547 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [LEADER]: Connected to new peer: Peer: permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 03:10:47.153106 28548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 03:10:47.154662 28549 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d2be03d77f4340b79c5056a541959897 (127.20.174.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:47.155826 28086 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d2be03d77f4340b79c5056a541959897 (127.20.174.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:47.156260 28549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:47.156461 28549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:47.156592 28549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:47.156683 28549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:47.157060 27847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:47.157181 27847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:47.157543 27903 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d2be03d77f4340b79c5056a541959897 (127.20.174.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:47.157814 28116 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:47.158008 28116 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:10:47.159536 28557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51489f943cd34cbb87a0d04a58ab918c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:47.162503 28557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:10:47.167025 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.161089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:44577 (local address 127.0.0.1:51454)
0929 03:10:47.161223 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:47.161238 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:47.161325 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:47.162087 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:47.162090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:47.162113 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:47.162314 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.162322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.164418 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.164422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.165619 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.165629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.166589 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.166593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.166681 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.166707 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:10:47.166773 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:10:47.166846 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":27}
I20260929 03:10:47.167052 28558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.161241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.20.174.65:44577)
0929 03:10:47.161501 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.161519 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.161542 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:10:47.161641 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.161649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.161900 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.162544 (+ 644us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:47.162686 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.164306 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.165750 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.166481 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.166588 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.166740 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.166807 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.166900 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":102,"threads_started":1}
I20260929 03:10:47.167382 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.161663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:35159 (local address 127.0.0.1:52120)
0929 03:10:47.162743 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:47.162755 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:47.162841 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:47.163564 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:47.163567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:47.163583 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:47.163773 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.163781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.164965 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.164970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.165809 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.165818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.166963 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.166967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.167054 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.167081 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 03:10:47.167133 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:10:47.167213 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":20,"thread_start_us":114,"threads_started":1}
I20260929 03:10:47.167989 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.161821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52120 (local address 127.20.174.67:35159)
0929 03:10:47.162388 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.162402 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.162852 (+ 450us) server_negotiation.cc:407] Connection header received
0929 03:10:47.162941 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.162946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.163208 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.163421 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:47.163902 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.164845 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.165940 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.166837 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.167541 (+ 704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.167697 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.167765 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.167856 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":125,"threads_started":1}
I20260929 03:10:47.168476 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.161673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:35397 (local address 127.0.0.1:51406)
0929 03:10:47.162972 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:47.162981 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:47.163080 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:47.163699 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:47.163702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:47.163717 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:47.163843 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.163852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.165187 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.165191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.166254 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.166264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.167774 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.167777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.168113 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.168148 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 03:10:47.168233 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:10:47.168321 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"thread_start_us":110,"threads_started":1}
I20260929 03:10:47.168578 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.162106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.20.174.66:35397)
0929 03:10:47.162492 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.162506 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.163074 (+ 568us) server_negotiation.cc:407] Connection header received
0929 03:10:47.163188 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.163194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.163419 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.163568 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:47.163981 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.165055 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.166390 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.167400 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.167948 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.168083 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.168281 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.168386 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260929 03:10:47.170744 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 344a4f24884d47498ca878fbdf255da7 (DEFAULT_TABLE table=test-table [id=ad10f5ce72bd4d94b4b894b1cf3c2555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:47.171165 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 344a4f24884d47498ca878fbdf255da7 (DEFAULT_TABLE table=test-table [id=ad10f5ce72bd4d94b4b894b1cf3c2555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:47.172004 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344a4f24884d47498ca878fbdf255da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:47.172145 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344a4f24884d47498ca878fbdf255da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:47.173331 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 344a4f24884d47498ca878fbdf255da7 (DEFAULT_TABLE table=test-table [id=ad10f5ce72bd4d94b4b894b1cf3c2555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:47.174259 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344a4f24884d47498ca878fbdf255da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:47.181157 28563 tablet_bootstrap.cc:492] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e: Bootstrap starting.
I20260929 03:10:47.181712 28565 tablet_bootstrap.cc:492] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf: Bootstrap starting.
I20260929 03:10:47.182850 28564 tablet_bootstrap.cc:492] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: Bootstrap starting.
I20260929 03:10:47.183353 28563 tablet_bootstrap.cc:654] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:47.183941 28565 tablet_bootstrap.cc:654] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:47.184211 28563 log.cc:824] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:47.184779 28565 log.cc:824] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:47.185029 28564 tablet_bootstrap.cc:654] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:47.185850 28564 log.cc:824] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:47.186480 28563 tablet_bootstrap.cc:492] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e: No bootstrap required, opened a new log
I20260929 03:10:47.186700 28563 ts_tablet_manager.cc:1397] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:10:47.187000 28565 tablet_bootstrap.cc:492] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf: No bootstrap required, opened a new log
I20260929 03:10:47.187181 28565 ts_tablet_manager.cc:1397] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 03:10:47.187939 28564 tablet_bootstrap.cc:492] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: No bootstrap required, opened a new log
I20260929 03:10:47.188125 28564 ts_tablet_manager.cc:1397] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:10:47.191490 28563 raft_consensus.cc:348] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:47.191504 28565 raft_consensus.cc:348] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:47.191772 28563 raft_consensus.cc:374] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:47.191777 28565 raft_consensus.cc:374] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:47.191829 28563 raft_consensus.cc:729] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c90b2b03d847c8a4a68151a867bb2e, State: Initialized, Role: FOLLOWER
I20260929 03:10:47.191829 28565 raft_consensus.cc:729] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aad6b620fd141db897db0570f975caf, State: Initialized, Role: FOLLOWER
I20260929 03:10:47.192320 28563 consensus_queue.cc:260] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:47.192333 28565 consensus_queue.cc:260] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:47.192900 28564 raft_consensus.cc:348] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:47.193115 28529 heartbeater.cc:503] Master 127.20.174.126:44627 was elected leader, sending a full tablet report...
I20260929 03:10:47.193176 28564 raft_consensus.cc:374] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:47.193240 28564 raft_consensus.cc:729] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23541f7f22ce42debe52d0985731995b, State: Initialized, Role: FOLLOWER
I20260929 03:10:47.193456 28393 heartbeater.cc:503] Master 127.20.174.126:44627 was elected leader, sending a full tablet report...
I20260929 03:10:47.193727 28564 consensus_queue.cc:260] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:47.194761 28564 ts_tablet_manager.cc:1428] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 03:10:47.195381 28256 heartbeater.cc:503] Master 127.20.174.126:44627 was elected leader, sending a full tablet report...
I20260929 03:10:47.197149 28565 ts_tablet_manager.cc:1428] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
W20260929 03:10:47.199198 28259 tablet.cc:2369] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:10:47.193471 28563 ts_tablet_manager.cc:1428] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260929 03:10:47.203379 28569 raft_consensus.cc:482] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:47.203534 28569 raft_consensus.cc:504] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:47.204746 28569 leader_election.cc:288] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23541f7f22ce42debe52d0985731995b (127.20.174.65:44577), 1aad6b620fd141db897db0570f975caf (127.20.174.67:35159)
I20260929 03:10:47.209868 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.204998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:35159 (local address 127.20.174.66:60553)
0929 03:10:47.205135 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:47.205146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:47.205200 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:47.205467 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:47.205470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:47.205486 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:47.205683 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.205690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.206812 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.206815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.208388 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.208401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.209332 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.209336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.209505 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.209531 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:10:47.209611 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:10:47.209692 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":186,"spinlock_wait_cycles":768}
I20260929 03:10:47.209873 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.205126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:60553 (local address 127.20.174.67:35159)
0929 03:10:47.205233 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.205238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.205255 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:10:47.205311 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.205314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.205359 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.205445 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:47.205839 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.206706 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.208551 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.209194 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.209319 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.209373 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.209647 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.209741 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 03:10:47.209897 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.204933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:44577 (local address 127.20.174.66:45181)
0929 03:10:47.205055 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:47.205070 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:47.205158 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:47.205491 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:47.205494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:47.205515 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:47.205687 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.205691 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.206812 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.206815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.208388 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:47.208401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.209437 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.209453 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:47.209572 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.209623 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 03:10:47.209697 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:10:47.209767 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":191,"spinlock_wait_cycles":1664}
I20260929 03:10:47.210300 28558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.205086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:45181 (local address 127.20.174.65:44577)
0929 03:10:47.205246 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.205251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.205263 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:10:47.205311 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.205314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.205359 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.205445 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:47.205895 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.206694 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.208551 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.209194 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.209319 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.209373 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.210025 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.210125 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 03:10:47.210997 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344a4f24884d47498ca878fbdf255da7" candidate_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aad6b620fd141db897db0570f975caf" is_pre_election: true
I20260929 03:10:47.211331 28483 raft_consensus.cc:2454] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c90b2b03d847c8a4a68151a867bb2e in term 0.
I20260929 03:10:47.211800 28282 leader_election.cc:302] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aad6b620fd141db897db0570f975caf, b0c90b2b03d847c8a4a68151a867bb2e; no voters:
I20260929 03:10:47.212124 28569 raft_consensus.cc:2781] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:47.212225 28569 raft_consensus.cc:482] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:47.212275 28569 raft_consensus.cc:3037] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:47.213670 28569 raft_consensus.cc:504] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:47.214061 28569 leader_election.cc:288] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [CANDIDATE]: Term 1 election: Requested vote from peers 23541f7f22ce42debe52d0985731995b (127.20.174.65:44577), 1aad6b620fd141db897db0570f975caf (127.20.174.67:35159)
I20260929 03:10:47.215071 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344a4f24884d47498ca878fbdf255da7" candidate_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aad6b620fd141db897db0570f975caf"
I20260929 03:10:47.215232 28483 raft_consensus.cc:3037] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:47.216359 28483 raft_consensus.cc:2454] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c90b2b03d847c8a4a68151a867bb2e in term 1.
I20260929 03:10:47.216679 28282 leader_election.cc:302] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aad6b620fd141db897db0570f975caf, b0c90b2b03d847c8a4a68151a867bb2e; no voters:
I20260929 03:10:47.217072 28569 raft_consensus.cc:2781] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:47.217315 28569 raft_consensus.cc:686] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 1 LEADER]: Becoming Leader. State: Replica: b0c90b2b03d847c8a4a68151a867bb2e, State: Running, Role: LEADER
I20260929 03:10:47.217677 28569 consensus_queue.cc:237] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:47.218638 28210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344a4f24884d47498ca878fbdf255da7" candidate_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23541f7f22ce42debe52d0985731995b" is_pre_election: true
I20260929 03:10:47.218719 28209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344a4f24884d47498ca878fbdf255da7" candidate_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23541f7f22ce42debe52d0985731995b"
I20260929 03:10:47.219005 28210 raft_consensus.cc:2454] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c90b2b03d847c8a4a68151a867bb2e in term 0.
I20260929 03:10:47.221874 27283 catalog_manager.cc:5495] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e reported cstate change: term changed from 0 to 1, leader changed from <none> to b0c90b2b03d847c8a4a68151a867bb2e (127.20.174.66). New cstate: current_term: 1 leader_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } health_report { overall_health: UNKNOWN } } }
I20260929 03:10:47.246573 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.242748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.20.174.66:35397)
0929 03:10:47.242884 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.242889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.242904 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:47.243040 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.243042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.243098 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.243213 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:47.243475 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.243994 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.244895 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.245148 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.245254 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.245591 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:47.246046 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:47.246192 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.246250 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.246314 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:10:47.253041 28209 raft_consensus.cc:3037] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:47.253206 28483 raft_consensus.cc:1261] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf [term 1 FOLLOWER]: Refusing update from remote peer b0c90b2b03d847c8a4a68151a867bb2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:47.254006 28572 consensus_queue.cc:1035] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:47.254702 28209 raft_consensus.cc:1261] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b [term 1 FOLLOWER]: Refusing update from remote peer b0c90b2b03d847c8a4a68151a867bb2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:47.255298 28572 consensus_queue.cc:1035] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:47.266930 28577 mvcc.cc:205] Tried to move back new op lower bound from 7334508327940370432 to 7334508327807135744. Current Snapshot: MvccSnapshot[applied={T|T < 7334508327940370432}]
I20260929 03:10:47.272097 28576 mvcc.cc:205] Tried to move back new op lower bound from 7334508327940370432 to 7334508327807135744. Current Snapshot: MvccSnapshot[applied={T|T < 7334508327940370432}]
I20260929 03:10:47.275290 27283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:47.275465 27283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:47.280676 27283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51489f943cd34cbb87a0d04a58ab918c: Sending DeleteTablet for 3 replicas of tablet 344a4f24884d47498ca878fbdf255da7
I20260929 03:10:47.281442 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet 344a4f24884d47498ca878fbdf255da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:47 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:51454
I20260929 03:10:47.281666 28463 tablet_service.cc:1514] Processing DeleteTablet for tablet 344a4f24884d47498ca878fbdf255da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:47 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:52120
I20260929 03:10:47.281692 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 344a4f24884d47498ca878fbdf255da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:47 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:51406
I20260929 03:10:47.282174 28586 tablet_replica.cc:330] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e: stopping tablet replica
I20260929 03:10:47.282408 28585 tablet_replica.cc:330] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: stopping tablet replica
I20260929 03:10:47.282577 28586 raft_consensus.cc:2229] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:47.282868 28585 raft_consensus.cc:2229] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:47.282989 28586 raft_consensus.cc:2258] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:47.283306 28585 raft_consensus.cc:2258] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:47.283380 28587 tablet_replica.cc:330] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf: stopping tablet replica
I20260929 03:10:47.283675 28587 raft_consensus.cc:2229] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:47.284006 28587 raft_consensus.cc:2258] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:47.285137 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 27253
I20260929 03:10:47.285138 28585 ts_tablet_manager.cc:1905] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:47.285297 28587 ts_tablet_manager.cc:1905] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:47.286355 28586 ts_tablet_manager.cc:1905] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:47.288551 28587 ts_tablet_manager.cc:1918] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:47.288551 28585 ts_tablet_manager.cc:1918] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:47.288662 28587 log.cc:1196] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/344a4f24884d47498ca878fbdf255da7
I20260929 03:10:47.288926 28585 log.cc:1196] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-0/wal/wals/344a4f24884d47498ca878fbdf255da7
I20260929 03:10:47.289016 28587 ts_tablet_manager.cc:1939] T 344a4f24884d47498ca878fbdf255da7 P 1aad6b620fd141db897db0570f975caf: Deleting consensus metadata
I20260929 03:10:47.289410 28585 ts_tablet_manager.cc:1939] T 344a4f24884d47498ca878fbdf255da7 P 23541f7f22ce42debe52d0985731995b: Deleting consensus metadata
W20260929 03:10:47.293798 28416 connection.cc:535] server connection from 127.0.0.1:52120 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 03:10:47.294221 28144 connection.cc:535] server connection from 127.0.0.1:51454 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:10:47.299260 28586 ts_tablet_manager.cc:1918] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:47.299392 28586 log.cc:1196] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/344a4f24884d47498ca878fbdf255da7
I20260929 03:10:47.299782 28586 ts_tablet_manager.cc:1939] T 344a4f24884d47498ca878fbdf255da7 P b0c90b2b03d847c8a4a68151a867bb2e: Deleting consensus metadata
W20260929 03:10:47.301584 28282 connection.cc:441] server connection from 127.0.0.1:51406 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51406 (request call id 1) could send its response
I20260929 03:10:47.303732 28541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.283849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.20.174.125:43575)
0929 03:10:47.284109 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.284114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.290744 (+ 6630us) server_negotiation.cc:407] Connection header received
0929 03:10:47.290956 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.290960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.291010 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.291103 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:47.291849 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.292388 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.293281 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.293530 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.293637 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.295781 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:47.295785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:47.295788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:47.295818 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:47.297600 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.298135 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.298138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.298140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.298223 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.298549 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.298551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.298553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.298777 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:47.303359 (+ 4582us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.303443 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.303569 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260929 03:10:47.306521 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.297300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.20.174.125:43575)
0929 03:10:47.297421 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.297426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.297855 (+ 429us) server_negotiation.cc:407] Connection header received
0929 03:10:47.297979 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.297982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.298035 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.298117 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:47.298869 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.299416 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.300412 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.300712 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.300803 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.302499 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:47.302503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:47.302506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:47.302530 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:47.304306 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.305130 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.305134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.305136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.305223 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.305517 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.305520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.305522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.305743 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:47.305877 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.306138 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.306259 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:10:47.308974 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.283980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.20.174.124:34329)
0929 03:10:47.284194 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.284198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.299660 (+ 15462us) server_negotiation.cc:407] Connection header received
0929 03:10:47.299715 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.299721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.299778 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.299859 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:47.300843 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.302144 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.303298 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.303701 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.303836 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.305491 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:47.305496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:47.305501 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:47.305532 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:47.307270 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.307668 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.307672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.307674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.307781 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.308090 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.308094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.308096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.308296 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:47.308441 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.308662 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.308791 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260929 03:10:47.316584 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:47.307827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.20.174.124:34329)
0929 03:10:47.307959 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:47.307963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:47.307976 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:10:47.308008 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:47.308011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:47.308048 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:47.308121 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:47.308854 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.309651 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.310971 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:47.311200 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:47.311352 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:47.312937 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:47.312940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:47.312942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:47.312962 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:47.314552 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.315165 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.315168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.315170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.315251 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:47.315576 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:47.315580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:47.315581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:47.315767 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:47.315961 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:47.316218 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 03:10:47.316340 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:10:48.269838 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.269262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.67:53213)
0929 03:10:48.269566 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.269715 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
W20260929 03:10:48.270182 28529 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:44627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
I20260929 03:10:48.272708 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.272256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.66:33493)
0929 03:10:48.272523 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.272617 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
W20260929 03:10:48.273023 28393 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:44627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
I20260929 03:10:48.273252 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.272718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.65:53917)
0929 03:10:48.273064 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.273165 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":107,"threads_started":1}
W20260929 03:10:48.273589 28256 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:44627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
I20260929 03:10:48.780287 28598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51489f943cd34cbb87a0d04a58ab918c)
I20260929 03:10:48.780464 28598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } }
I20260929 03:10:48.780512 28599 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51489f943cd34cbb87a0d04a58ab918c)
I20260929 03:10:48.780722 28599 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } }
I20260929 03:10:48.781147 28598 leader_election.cc:288] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627), d2be03d77f4340b79c5056a541959897 (127.20.174.124:34329)
I20260929 03:10:48.781908 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.781305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:44328)
0929 03:10:48.781680 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.781789 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":84,"threads_started":1}
I20260929 03:10:48.781986 28599 leader_election.cc:288] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627), 573016e12e7e4aa9b3e8161371367e69 (127.20.174.125:43575)
W20260929 03:10:48.782271 27879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
I20260929 03:10:48.782578 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.782096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:44344)
0929 03:10:48.782385 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.782484 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":55,"threads_started":1}
W20260929 03:10:48.782732 28063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
I20260929 03:10:48.783668 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.783339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:44356)
0929 03:10:48.783557 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.783607 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260929 03:10:48.783776 28063 leader_election.cc:334] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627): Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
I20260929 03:10:48.784145 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.783410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:44372)
0929 03:10:48.784032 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.784081 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559}
W20260929 03:10:48.784255 27879 leader_election.cc:334] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627): Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
I20260929 03:10:48.786162 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.781398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:34329 (local address 127.0.0.1:40286)
0929 03:10:48.781702 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.781715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:48.781820 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:48.782192 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:48.782195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:48.782216 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:48.782335 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:48.782342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:48.783902 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:48.783906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:48.784960 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:48.784971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:48.785685 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:48.785688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:48.785830 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:48.785870 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:10:48.785969 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 03:10:48.786046 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":32,"thread_start_us":61,"threads_started":1}
I20260929 03:10:48.786554 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.782392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:43575 (local address 127.0.0.1:33894)
0929 03:10:48.782625 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.782640 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:48.782731 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:48.783192 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:48.783195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:48.783214 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:48.783331 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:48.783338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:48.784317 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:48.784320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:48.785159 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:48.785170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:48.786053 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:48.786059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:48.786152 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:48.786172 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:10:48.786278 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:10:48.786359 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260929 03:10:48.786648 28606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.782655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.20.174.125:43575)
0929 03:10:48.782939 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:48.782945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:48.782957 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:10:48.783019 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:48.783022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:48.783066 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:48.783436 (+ 370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:48.783552 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:48.784168 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:48.785294 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:48.785880 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:48.785992 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:48.786030 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:48.786412 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 03:10:48.786533 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260929 03:10:48.786785 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.781587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.20.174.124:34329)
0929 03:10:48.781807 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:48.781812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:48.781829 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:10:48.781974 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:48.781979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:48.782034 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:48.782853 (+ 819us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:48.782976 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:48.783749 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:48.785089 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:48.785548 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:48.785679 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:48.785740 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:48.786524 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 03:10:48.786647 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260929 03:10:48.787189 28086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "573016e12e7e4aa9b3e8161371367e69" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d2be03d77f4340b79c5056a541959897" is_pre_election: true
I20260929 03:10:48.787202 27903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2be03d77f4340b79c5056a541959897" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "573016e12e7e4aa9b3e8161371367e69" is_pre_election: true
I20260929 03:10:48.787410 28086 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573016e12e7e4aa9b3e8161371367e69 in term 1.
I20260929 03:10:48.787413 27903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2be03d77f4340b79c5056a541959897 in term 1.
I20260929 03:10:48.787770 27878 leader_election.cc:302] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 573016e12e7e4aa9b3e8161371367e69, d2be03d77f4340b79c5056a541959897; no voters: 51489f943cd34cbb87a0d04a58ab918c
I20260929 03:10:48.787783 28064 leader_election.cc:302] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 573016e12e7e4aa9b3e8161371367e69, d2be03d77f4340b79c5056a541959897; no voters: 51489f943cd34cbb87a0d04a58ab918c
I20260929 03:10:48.787940 28598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 03:10:48.787993 28598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 51489f943cd34cbb87a0d04a58ab918c)
I20260929 03:10:48.787983 28599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 03:10:48.788034 28598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:10:48.788061 28599 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 51489f943cd34cbb87a0d04a58ab918c)
I20260929 03:10:48.788105 28599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:10:48.789048 28598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } }
I20260929 03:10:48.789081 28599 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } }
I20260929 03:10:48.789407 28598 leader_election.cc:288] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [CANDIDATE]: Term 2 election: Requested vote from peers 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627), d2be03d77f4340b79c5056a541959897 (127.20.174.124:34329)
I20260929 03:10:48.789470 28599 leader_election.cc:288] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 2 election: Requested vote from peers 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627), 573016e12e7e4aa9b3e8161371367e69 (127.20.174.125:43575)
I20260929 03:10:48.789901 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.789515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:44382)
0929 03:10:48.789772 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.789829 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260929 03:10:48.789862 27903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2be03d77f4340b79c5056a541959897" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "573016e12e7e4aa9b3e8161371367e69"
I20260929 03:10:48.790117 27903 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d2be03d77f4340b79c5056a541959897 in current term 2: Already voted for candidate 573016e12e7e4aa9b3e8161371367e69 in this term.
I20260929 03:10:48.790120 28086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "573016e12e7e4aa9b3e8161371367e69" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d2be03d77f4340b79c5056a541959897"
I20260929 03:10:48.790284 28086 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 573016e12e7e4aa9b3e8161371367e69 in current term 2: Already voted for candidate d2be03d77f4340b79c5056a541959897 in this term.
I20260929 03:10:48.790645 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.789531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:44390)
0929 03:10:48.790511 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.790579 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":915}
I20260929 03:10:48.790649 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.790398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:44398)
0929 03:10:48.790534 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.790588 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 03:10:48.790750 27879 leader_election.cc:334] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627): Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
I20260929 03:10:48.790819 27879 leader_election.cc:302] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 573016e12e7e4aa9b3e8161371367e69; no voters: 51489f943cd34cbb87a0d04a58ab918c, d2be03d77f4340b79c5056a541959897
I20260929 03:10:48.790979 28598 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 03:10:48.791352 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:48.791165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:44404)
0929 03:10:48.791251 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:48.791301 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:10:48.791481 28063 leader_election.cc:334] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627): Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
I20260929 03:10:48.791558 28063 leader_election.cc:302] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2be03d77f4340b79c5056a541959897; no voters: 51489f943cd34cbb87a0d04a58ab918c, 573016e12e7e4aa9b3e8161371367e69
I20260929 03:10:48.791705 28599 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 03:10:49.271287 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:49.270769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.67:51183)
0929 03:10:49.271100 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:49.271162 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260929 03:10:49.274027 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:49.273553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.66:43211)
0929 03:10:49.273814 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:49.273904 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260929 03:10:49.274585 28610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:49.274093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.65:47857)
0929 03:10:49.274374 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:49.274505 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":76,"threads_started":1}
I20260929 03:10:50.272388 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.271968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.67:51023)
0929 03:10:50.272230 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.272298 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260929 03:10:50.275249 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.274745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.66:50207)
0929 03:10:50.275068 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.275142 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":85,"threads_started":1}
I20260929 03:10:50.275676 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.275230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.65:32803)
0929 03:10:50.275556 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.275602 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":81,"threads_started":1}
I20260929 03:10:50.594844 28614 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:50.594980 28614 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } }
I20260929 03:10:50.595440 28614 leader_election.cc:288] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627), 573016e12e7e4aa9b3e8161371367e69 (127.20.174.125:43575)
I20260929 03:10:50.596104 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.595680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:34666)
0929 03:10:50.595922 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.595995 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260929 03:10:50.596042 27903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2be03d77f4340b79c5056a541959897" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "573016e12e7e4aa9b3e8161371367e69" is_pre_election: true
I20260929 03:10:50.596206 27903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2be03d77f4340b79c5056a541959897 in term 2.
I20260929 03:10:50.596597 28064 leader_election.cc:302] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573016e12e7e4aa9b3e8161371367e69, d2be03d77f4340b79c5056a541959897; no voters:
I20260929 03:10:50.596755 28614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 03:10:50.596808 28614 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:50.596868 28614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:10:50.597115 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.596851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:34676)
0929 03:10:50.597013 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.597058 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260929 03:10:50.597252 28063 leader_election.cc:334] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627): Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
I20260929 03:10:50.598040 28614 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } }
I20260929 03:10:50.598398 28614 leader_election.cc:288] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 3 election: Requested vote from peers 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627), 573016e12e7e4aa9b3e8161371367e69 (127.20.174.125:43575)
I20260929 03:10:50.598737 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.598539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:34684)
0929 03:10:50.598625 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.598681 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 03:10:50.598929 27903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2be03d77f4340b79c5056a541959897" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "573016e12e7e4aa9b3e8161371367e69"
I20260929 03:10:50.599089 27903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:10:50.599387 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.599202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:34686)
0929 03:10:50.599286 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.599337 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260929 03:10:50.599490 28063 leader_election.cc:334] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627): Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
I20260929 03:10:50.600468 27903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2be03d77f4340b79c5056a541959897 in term 3.
I20260929 03:10:50.600816 28064 leader_election.cc:302] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 573016e12e7e4aa9b3e8161371367e69, d2be03d77f4340b79c5056a541959897; no voters: 51489f943cd34cbb87a0d04a58ab918c
I20260929 03:10:50.601015 28614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:10:50.601210 28614 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [term 3 LEADER]: Becoming Leader. State: Replica: d2be03d77f4340b79c5056a541959897, State: Running, Role: LEADER
I20260929 03:10:50.601480 28614 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } }
I20260929 03:10:50.602922 28617 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2be03d77f4340b79c5056a541959897. Latest consensus state: current_term: 3 leader_uuid: "d2be03d77f4340b79c5056a541959897" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:50.603068 28617 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:50.603400 28619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:50.606340 28619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:50.606942 28619 catalog_manager.cc:1259] Loaded cluster ID: f9cd3cde7b3b45f2a0a453c7770a66f0
I20260929 03:10:50.607002 28619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:50.607569 28619 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:50.607983 28619 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: Loaded TSK: 0
I20260929 03:10:50.608706 28619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:10:50.699324 28076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:50.702595 28076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:50.705897 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.705606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:34710)
0929 03:10:50.705740 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.705817 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 03:10:50.706044 27903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [term 3 FOLLOWER]: Refusing update from remote peer d2be03d77f4340b79c5056a541959897: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 03:10:50.706597 28617 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [LEADER]: Connected to new peer: Peer: permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 03:10:50.706759 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.706511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.0.0.1:34726)
0929 03:10:50.706618 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.706674 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:10:50.706895 28063 consensus_peers.cc:487] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 -> Peer 51489f943cd34cbb87a0d04a58ab918c (127.20.174.126:44627): Couldn't send request to peer 51489f943cd34cbb87a0d04a58ab918c. Status: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 03:10:50.708216 28621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2be03d77f4340b79c5056a541959897. Latest consensus state: current_term: 3 leader_uuid: "d2be03d77f4340b79c5056a541959897" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:50.708385 28621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:10:50.709635 28614 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d2be03d77f4340b79c5056a541959897" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:50.709806 28614 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:50.710065 28621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d2be03d77f4340b79c5056a541959897" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "51489f943cd34cbb87a0d04a58ab918c" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44627 } } peers { permanent_uuid: "573016e12e7e4aa9b3e8161371367e69" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 43575 } attrs { promote: false } } peers { permanent_uuid: "d2be03d77f4340b79c5056a541959897" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 34329 } attrs { promote: false } } }
I20260929 03:10:50.710196 28621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573016e12e7e4aa9b3e8161371367e69 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:10:50.721796 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.717258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:35397 (local address 127.0.0.1:54744)
0929 03:10:50.717366 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.717383 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:50.717478 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:50.718200 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:50.718206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:50.718230 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:50.718359 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:50.718367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.719256 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.719260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:50.720538 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:50.720547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.721364 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.721367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:50.721471 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:50.721494 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:10:50.721592 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 03:10:50.721675 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":27,"spinlock_wait_cycles":768}
I20260929 03:10:50.722383 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.717358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.20.174.66:35397)
0929 03:10:50.717597 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:50.717601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:50.717615 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:10:50.717674 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:50.717680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:50.717736 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:50.717829 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:50.718512 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.719085 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.720691 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.721193 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.721357 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:50.721429 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:50.722080 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 03:10:50.722177 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
I20260929 03:10:50.723016 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebbf7d84da44e438bcbe3bd7f35a588 (DEFAULT_TABLE table=test-table [id=abecf534385a45a69ffd4bde43b86f0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:50.723341 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebbf7d84da44e438bcbe3bd7f35a588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:50.723645 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.718482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.20.174.67:35159)
0929 03:10:50.718703 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:50.718708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:50.718787 (+ 79us) server_negotiation.cc:407] Connection header received
0929 03:10:50.718906 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:50.718971 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:50.719032 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:50.719122 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:50.719508 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.720310 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.722152 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.722881 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.723000 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:50.723068 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:50.723431 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 03:10:50.723512 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260929 03:10:50.723664 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.717857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:44577 (local address 127.0.0.1:44042)
0929 03:10:50.718180 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.718195 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:50.718290 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:50.719446 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:50.719449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:50.719465 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:50.719540 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:50.719546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.720639 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.720642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:50.722016 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:50.722028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.723260 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.723263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:50.723398 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:50.723421 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:10:50.723467 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:10:50.723522 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":3,"spinlock_wait_cycles":2304,"thread_start_us":116,"threads_started":1}
I20260929 03:10:50.723670 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.718118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:35159 (local address 127.0.0.1:43176)
0929 03:10:50.718722 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.718735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:50.718822 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:50.719160 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:50.719167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:50.719193 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:50.719350 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:50.719358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.720489 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.720492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:50.721966 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:50.721978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.723134 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.723138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:50.723255 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:50.723320 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 03:10:50.723397 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:10:50.723488 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":94,"thread_start_us":72,"threads_started":1}
I20260929 03:10:50.724231 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebbf7d84da44e438bcbe3bd7f35a588 (DEFAULT_TABLE table=test-table [id=abecf534385a45a69ffd4bde43b86f0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:50.724361 28629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.717996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.20.174.65:44577)
0929 03:10:50.719035 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:50.719039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:50.719054 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:50.719110 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:50.719113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:50.719155 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:50.719645 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:50.719750 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.720536 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.722209 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.723129 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.724004 (+ 875us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:50.724075 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:50.724116 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 03:10:50.724196 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":83,"threads_started":1}
I20260929 03:10:50.724545 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebbf7d84da44e438bcbe3bd7f35a588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:50.724849 28190 tablet_service.cc:1467] Processing CreateTablet for tablet 8ebbf7d84da44e438bcbe3bd7f35a588 (DEFAULT_TABLE table=test-table [id=abecf534385a45a69ffd4bde43b86f0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:50.725114 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ebbf7d84da44e438bcbe3bd7f35a588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:50.725633 28632 tablet_bootstrap.cc:492] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e: Bootstrap starting.
I20260929 03:10:50.727020 28632 tablet_bootstrap.cc:654] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:50.727597 28633 tablet_bootstrap.cc:492] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf: Bootstrap starting.
I20260929 03:10:50.727914 28635 tablet_bootstrap.cc:492] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b: Bootstrap starting.
I20260929 03:10:50.728716 28633 tablet_bootstrap.cc:654] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:50.728740 28635 tablet_bootstrap.cc:654] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:50.730014 28632 tablet_bootstrap.cc:492] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e: No bootstrap required, opened a new log
I20260929 03:10:50.730053 28633 tablet_bootstrap.cc:492] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf: No bootstrap required, opened a new log
I20260929 03:10:50.730123 28633 ts_tablet_manager.cc:1397] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:50.730123 28632 ts_tablet_manager.cc:1397] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 03:10:50.730579 28632 raft_consensus.cc:348] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:50.730579 28633 raft_consensus.cc:348] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:50.730706 28632 raft_consensus.cc:374] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:50.730706 28633 raft_consensus.cc:374] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:50.730742 28632 raft_consensus.cc:729] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c90b2b03d847c8a4a68151a867bb2e, State: Initialized, Role: FOLLOWER
I20260929 03:10:50.730742 28633 raft_consensus.cc:729] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aad6b620fd141db897db0570f975caf, State: Initialized, Role: FOLLOWER
I20260929 03:10:50.730885 28633 consensus_queue.cc:260] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:50.730937 28632 consensus_queue.cc:260] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:50.731065 28635 tablet_bootstrap.cc:492] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b: No bootstrap required, opened a new log
I20260929 03:10:50.731159 28635 ts_tablet_manager.cc:1397] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:50.731277 28633 ts_tablet_manager.cc:1428] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:50.731277 28632 ts_tablet_manager.cc:1428] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:50.731652 28635 raft_consensus.cc:348] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:50.731786 28635 raft_consensus.cc:374] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:50.731820 28635 raft_consensus.cc:729] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23541f7f22ce42debe52d0985731995b, State: Initialized, Role: FOLLOWER
I20260929 03:10:50.731949 28635 consensus_queue.cc:260] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:50.732245 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.731745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.66:39233)
0929 03:10:50.732107 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.732171 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282}
I20260929 03:10:50.732308 28635 ts_tablet_manager.cc:1428] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 03:10:50.733824 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.733555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.65:40953)
0929 03:10:50.733683 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.733746 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 03:10:50.734076 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.733798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.67:43937)
0929 03:10:50.733926 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.733990 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 03:10:50.735486 28530 heartbeater.cc:503] Master 127.20.174.124:34329 was elected leader, sending a full tablet report...
I20260929 03:10:50.735594 28394 heartbeater.cc:503] Master 127.20.174.124:34329 was elected leader, sending a full tablet report...
I20260929 03:10:50.735627 28257 heartbeater.cc:503] Master 127.20.174.124:34329 was elected leader, sending a full tablet report...
I20260929 03:10:50.736263 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.736000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.65:51969)
0929 03:10:50.736144 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.736193 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 03:10:50.736299 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.736055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.66:57079)
0929 03:10:50.736195 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.736239 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 03:10:50.736352 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.735723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.67:59155)
0929 03:10:50.736215 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.736275 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420}
W20260929 03:10:50.868651 28532 tablet.cc:2369] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:10:50.894515 28640 raft_consensus.cc:482] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:50.894685 28640 raft_consensus.cc:504] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:50.895851 28640 leader_election.cc:288] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0c90b2b03d847c8a4a68151a867bb2e (127.20.174.66:35397), 1aad6b620fd141db897db0570f975caf (127.20.174.67:35159)
I20260929 03:10:50.899933 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.896181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:35159 (local address 127.20.174.65:44387)
0929 03:10:50.896396 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.896412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:50.896575 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:50.896861 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:50.896864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:50.896882 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:50.897017 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:50.897022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.897758 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.897763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:50.898786 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:50.898796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.899507 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.899510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:50.899616 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:50.899641 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:10:50.899714 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:10:50.899799 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260929 03:10:50.899971 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.896243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:44387 (local address 127.20.174.67:35159)
0929 03:10:50.896386 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:50.896391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:50.896510 (+ 119us) server_negotiation.cc:407] Connection header received
0929 03:10:50.896704 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:50.896707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:50.896756 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:50.896838 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:50.897126 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.897647 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.898920 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.899341 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.899494 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:50.899548 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:50.899749 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 03:10:50.899845 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:10:50.900602 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebbf7d84da44e438bcbe3bd7f35a588" candidate_uuid: "23541f7f22ce42debe52d0985731995b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aad6b620fd141db897db0570f975caf" is_pre_election: true
I20260929 03:10:50.900693 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.896267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:35397 (local address 127.20.174.65:57645)
0929 03:10:50.896680 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.896691 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:50.896833 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:50.897090 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:50.897093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:50.897109 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:50.897218 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:50.897225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.898015 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.898019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:50.899303 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:50.899314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.900265 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.900268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:50.900368 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:50.900389 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:10:50.900471 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:10:50.900549 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":154,"threads_started":1}
I20260929 03:10:50.900714 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.896362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:57645 (local address 127.20.174.66:35397)
0929 03:10:50.896525 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:50.896529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:50.896789 (+ 260us) server_negotiation.cc:407] Connection header received
0929 03:10:50.896943 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:50.896946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:50.896994 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:50.897087 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:50.897328 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.897878 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.899451 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.900022 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.900145 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:50.900203 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:50.900522 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 03:10:50.900613 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:10:50.900784 28483 raft_consensus.cc:2454] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23541f7f22ce42debe52d0985731995b in term 0.
I20260929 03:10:50.901211 28145 leader_election.cc:302] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aad6b620fd141db897db0570f975caf, 23541f7f22ce42debe52d0985731995b; no voters:
I20260929 03:10:50.901520 28640 raft_consensus.cc:2781] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:50.901475 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebbf7d84da44e438bcbe3bd7f35a588" candidate_uuid: "23541f7f22ce42debe52d0985731995b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" is_pre_election: true
I20260929 03:10:50.901582 28640 raft_consensus.cc:482] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:50.901623 28640 raft_consensus.cc:3037] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:50.901715 28347 raft_consensus.cc:2454] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23541f7f22ce42debe52d0985731995b in term 0.
I20260929 03:10:50.902937 28640 raft_consensus.cc:504] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:50.903344 28640 leader_election.cc:288] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [CANDIDATE]: Term 1 election: Requested vote from peers b0c90b2b03d847c8a4a68151a867bb2e (127.20.174.66:35397), 1aad6b620fd141db897db0570f975caf (127.20.174.67:35159)
I20260929 03:10:50.903676 28347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebbf7d84da44e438bcbe3bd7f35a588" candidate_uuid: "23541f7f22ce42debe52d0985731995b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c90b2b03d847c8a4a68151a867bb2e"
I20260929 03:10:50.903753 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ebbf7d84da44e438bcbe3bd7f35a588" candidate_uuid: "23541f7f22ce42debe52d0985731995b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aad6b620fd141db897db0570f975caf"
I20260929 03:10:50.903800 28347 raft_consensus.cc:3037] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:50.903879 28483 raft_consensus.cc:3037] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf [term 0 FOLLOWER]: Advancing to term 1
W20260929 03:10:50.903980 28396 tablet.cc:2369] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:10:50.904981 28483 raft_consensus.cc:2454] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23541f7f22ce42debe52d0985731995b in term 1.
I20260929 03:10:50.904981 28347 raft_consensus.cc:2454] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23541f7f22ce42debe52d0985731995b in term 1.
I20260929 03:10:50.905282 28145 leader_election.cc:302] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aad6b620fd141db897db0570f975caf, 23541f7f22ce42debe52d0985731995b; no voters:
I20260929 03:10:50.905447 28640 raft_consensus.cc:2781] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:50.905671 28640 raft_consensus.cc:686] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 1 LEADER]: Becoming Leader. State: Replica: 23541f7f22ce42debe52d0985731995b, State: Running, Role: LEADER
I20260929 03:10:50.905946 28640 consensus_queue.cc:237] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } } peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } }
I20260929 03:10:50.906597 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.906159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.65:50289)
0929 03:10:50.906466 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.906526 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260929 03:10:50.908710 28076 catalog_manager.cc:5495] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b reported cstate change: term changed from 0 to 1, leader changed from <none> to 23541f7f22ce42debe52d0985731995b (127.20.174.65). New cstate: current_term: 1 leader_uuid: "23541f7f22ce42debe52d0985731995b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23541f7f22ce42debe52d0985731995b" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 } health_report { overall_health: UNKNOWN } } }
I20260929 03:10:50.962945 28629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.958280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.20.174.65:44577)
0929 03:10:50.958409 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:50.958414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:50.958459 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:10:50.958556 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:50.958558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:50.958614 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:50.958728 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:50.958980 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.959860 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.961082 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:50.961466 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:50.961571 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:50.961826 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:50.962371 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:50.962572 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:50.962736 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 03:10:50.962797 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:10:50.965794 28483 raft_consensus.cc:1261] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf [term 1 FOLLOWER]: Refusing update from remote peer 23541f7f22ce42debe52d0985731995b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:50.966064 28347 raft_consensus.cc:1261] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e [term 1 FOLLOWER]: Refusing update from remote peer 23541f7f22ce42debe52d0985731995b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:50.966346 28640 consensus_queue.cc:1035] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aad6b620fd141db897db0570f975caf" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 35159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:50.966643 28642 consensus_queue.cc:1035] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c90b2b03d847c8a4a68151a867bb2e" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:50.968552 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.968025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.67:38169)
0929 03:10:50.968394 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.968463 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297}
I20260929 03:10:50.970870 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.969688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.66:42079)
0929 03:10:50.970706 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.970770 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":939}
I20260929 03:10:50.971688 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.971400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.65:52005)
0929 03:10:50.971545 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.971599 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 03:10:50.973287 28647 mvcc.cc:205] Tried to move back new op lower bound from 7334508343151026176 to 7334508342913900544. Current Snapshot: MvccSnapshot[applied={T|T < 7334508343151026176 or (T in {7334508343151026176})}]
I20260929 03:10:50.973579 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.973284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.66:54417)
0929 03:10:50.973440 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.973504 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 03:10:50.977488 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.977235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.65:36197)
0929 03:10:50.977342 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.977405 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 03:10:50.979219 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.978919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44627 (local address 127.20.174.67:55769)
0929 03:10:50.979078 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.979135 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 03:10:50.981619 28077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:50.981771 28077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:50.986009 28077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2be03d77f4340b79c5056a541959897: Sending DeleteTablet for 3 replicas of tablet 8ebbf7d84da44e438bcbe3bd7f35a588
I20260929 03:10:50.986809 28190 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ebbf7d84da44e438bcbe3bd7f35a588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:50 UTC) from {username='oryx', principal='oryx/127.20.174.124@KRBTEST.COM'} at 127.0.0.1:44042
I20260929 03:10:50.986963 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ebbf7d84da44e438bcbe3bd7f35a588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:50 UTC) from {username='oryx', principal='oryx/127.20.174.124@KRBTEST.COM'} at 127.0.0.1:54744
I20260929 03:10:50.987254 28463 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ebbf7d84da44e438bcbe3bd7f35a588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:50 UTC) from {username='oryx', principal='oryx/127.20.174.124@KRBTEST.COM'} at 127.0.0.1:43176
I20260929 03:10:50.987339 28656 tablet_replica.cc:330] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e: stopping tablet replica
I20260929 03:10:50.987327 28655 tablet_replica.cc:330] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b: stopping tablet replica
I20260929 03:10:50.987464 28656 raft_consensus.cc:2229] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:50.987464 28655 raft_consensus.cc:2229] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:50.987591 28656 raft_consensus.cc:2258] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:50.987803 28655 raft_consensus.cc:2258] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 23541f7f22ce42debe52d0985731995b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:50.988404 28656 ts_tablet_manager.cc:1905] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:50.988574 28657 tablet_replica.cc:330] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf: stopping tablet replica
I20260929 03:10:50.988755 28657 raft_consensus.cc:2229] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:50.988976 28657 raft_consensus.cc:2258] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:50.989137 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 28038
I20260929 03:10:50.990780 28657 ts_tablet_manager.cc:1905] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:50.992195 28656 ts_tablet_manager.cc:1918] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:50.992292 28656 log.cc:1196] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/8ebbf7d84da44e438bcbe3bd7f35a588
I20260929 03:10:50.992632 28656 ts_tablet_manager.cc:1939] T 8ebbf7d84da44e438bcbe3bd7f35a588 P b0c90b2b03d847c8a4a68151a867bb2e: Deleting consensus metadata
I20260929 03:10:50.993196 28657 ts_tablet_manager.cc:1918] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:50.993290 28657 log.cc:1196] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/8ebbf7d84da44e438bcbe3bd7f35a588
I20260929 03:10:50.993556 28062 catalog_manager.cc:4841] TS b0c90b2b03d847c8a4a68151a867bb2e (127.20.174.66:35397): tablet 8ebbf7d84da44e438bcbe3bd7f35a588 (table test-table [id=abecf534385a45a69ffd4bde43b86f0d]) successfully deleted
I20260929 03:10:50.993734 28657 ts_tablet_manager.cc:1939] T 8ebbf7d84da44e438bcbe3bd7f35a588 P 1aad6b620fd141db897db0570f975caf: Deleting consensus metadata
I20260929 03:10:50.994360 28063 catalog_manager.cc:4841] TS 1aad6b620fd141db897db0570f975caf (127.20.174.67:35159): tablet 8ebbf7d84da44e438bcbe3bd7f35a588 (table test-table [id=abecf534385a45a69ffd4bde43b86f0d]) successfully deleted
I20260929 03:10:50.997594 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 28264
I20260929 03:10:50.997720 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:50.997399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:44577 (local address 127.0.0.1:44056)
0929 03:10:50.997557 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:50.997618 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:44577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 03:10:50.998623 28061 catalog_manager.cc:4584] TS 23541f7f22ce42debe52d0985731995b (127.20.174.65:44577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ebbf7d84da44e438bcbe3bd7f35a588: Network error: Client connection negotiation failed: client connection to 127.20.174.65:44577: connect: Connection refused (error 111)
I20260929 03:10:51.004670 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 28400
I20260929 03:10:51.010746 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 27749
I20260929 03:10:51.017004 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 27859
2026-09-29T03:10:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7392 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:51 dist-test-slave-k1dg krb5kdc[28660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:51 dist-test-slave-k1dg krb5kdc[28660](info): set up 2 sockets
Sep 29 03:10:51 dist-test-slave-k1dg krb5kdc[28660](info): commencing operation
krb5kdc: starting...
W20260929 03:10:52.083161 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:52 dist-test-slave-k1dg krb5kdc[28660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.126.keytab.
Entry for principal oryx/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.126.keytab.
I20260929 03:10:52.216151 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44895
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.126.keytab
--principal=oryx/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:44895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:52.330061 28673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:52.330415 28673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:52.330526 28673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:52.334712 28673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:52.334820 28673 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:10:52.334851 28673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:52.334903 28673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:52.334944 28673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:52.334982 28673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:52.340588 28673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:44895
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.126.keytab
--principal=oryx/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:52.341759 28673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:52.342979 28673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:52.351653 28681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:52.351830 28678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:52.351884 28679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:52.352046 28673 server_base.cc:1034] running on GCE node
I20260929 03:10:52.352685 28673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:52.353896 28673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:52.355088 28673 hybrid_clock.cc:648] HybridClock initialized: now 1790651452355076 us; error 33 us; skew 500 ppm
Sep 29 03:10:52 dist-test-slave-k1dg krb5kdc[28660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651452, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:52.357995 28673 init.cc:377] Logged in from keytab as oryx/127.20.174.126@KRBTEST.COM (short username oryx)
I20260929 03:10:52.359062 28673 webserver.cc:460] Webserver started at http://127.20.174.126:34269/ using document root <none> and password file <none>
I20260929 03:10:52.359607 28673 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:52.359666 28673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:52.359833 28673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:52.361840 28673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "f81152bdbe6b4d54bfa9ca02161e0267"
format_stamp: "Formatted at 2026-09-29 03:10:52 on dist-test-slave-k1dg"
server_key: "aeabe33d60f9f1f077e6290a3be057c9"
server_key_iv: "069391abb5df0ad7dc57ea123d700558"
server_key_version: "encryptionkey@0"
I20260929 03:10:52.362358 28673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "f81152bdbe6b4d54bfa9ca02161e0267"
format_stamp: "Formatted at 2026-09-29 03:10:52 on dist-test-slave-k1dg"
server_key: "aeabe33d60f9f1f077e6290a3be057c9"
server_key_iv: "069391abb5df0ad7dc57ea123d700558"
server_key_version: "encryptionkey@0"
I20260929 03:10:52.366308 28673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:10:52.368988 28688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:52.370323 28673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:10:52.370532 28673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "f81152bdbe6b4d54bfa9ca02161e0267"
format_stamp: "Formatted at 2026-09-29 03:10:52 on dist-test-slave-k1dg"
server_key: "aeabe33d60f9f1f077e6290a3be057c9"
server_key_iv: "069391abb5df0ad7dc57ea123d700558"
server_key_version: "encryptionkey@0"
I20260929 03:10:52.370646 28673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:52.392086 28673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:52.395632 28673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:52.395805 28673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:52.404397 28673 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:44895
I20260929 03:10:52.404420 28750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:44895 every 8 connection(s)
I20260929 03:10:52.405438 28673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:52.408483 28751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:52.413360 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 28673
I20260929 03:10:52.413501 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:52.413776 21177 external_mini_cluster.cc:1428] Setting key 8481c9174ad3dbda5dcc032011ca7de3
I20260929 03:10:52.414920 28751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267: Bootstrap starting.
I20260929 03:10:52.417357 28751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:52.418135 28751 log.cc:824] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:52.420204 28751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267: No bootstrap required, opened a new log
Sep 29 03:10:52 dist-test-slave-k1dg krb5kdc[28660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:52.425239 28751 raft_consensus.cc:348] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81152bdbe6b4d54bfa9ca02161e0267" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44895 } }
I20260929 03:10:52.425596 28751 raft_consensus.cc:374] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:52.425670 28751 raft_consensus.cc:729] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81152bdbe6b4d54bfa9ca02161e0267, State: Initialized, Role: FOLLOWER
I20260929 03:10:52.426404 28751 consensus_queue.cc:260] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81152bdbe6b4d54bfa9ca02161e0267" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44895 } }
I20260929 03:10:52.426635 28751 raft_consensus.cc:388] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:52.426724 28751 raft_consensus.cc:482] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:52.426839 28751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:52.428277 28751 raft_consensus.cc:504] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81152bdbe6b4d54bfa9ca02161e0267" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44895 } }
I20260929 03:10:52.428684 28751 leader_election.cc:302] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f81152bdbe6b4d54bfa9ca02161e0267; no voters:
I20260929 03:10:52.429065 28751 leader_election.cc:288] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:52.429292 28756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:52.429526 28756 raft_consensus.cc:686] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [term 1 LEADER]: Becoming Leader. State: Replica: f81152bdbe6b4d54bfa9ca02161e0267, State: Running, Role: LEADER
I20260929 03:10:52.429859 28756 consensus_queue.cc:237] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81152bdbe6b4d54bfa9ca02161e0267" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44895 } }
I20260929 03:10:52.430346 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:52.415333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.20.174.126:44895)
0929 03:10:52.415813 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:52.415829 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:52.415868 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:10:52.416591 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:52.416611 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:52.416966 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:52.417355 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:52.418189 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.419445 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.420409 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.421111 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.421315 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:52.424244 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:52.424251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:52.424271 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:52.424311 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:52.426548 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:52.427214 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:52.427220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:52.427226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:52.427380 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:52.427692 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:52.427695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:52.427697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:52.428418 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:52.428625 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:52.428872 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 03:10:52.429250 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":148,"threads_started":1}
I20260929 03:10:52.430475 28751 sys_catalog.cc:564] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:52.431807 28758 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f81152bdbe6b4d54bfa9ca02161e0267. Latest consensus state: current_term: 1 leader_uuid: "f81152bdbe6b4d54bfa9ca02161e0267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81152bdbe6b4d54bfa9ca02161e0267" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44895 } } }
I20260929 03:10:52.431807 28757 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f81152bdbe6b4d54bfa9ca02161e0267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81152bdbe6b4d54bfa9ca02161e0267" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44895 } } }
I20260929 03:10:52.431967 28758 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:52.431967 28757 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267 [sys.catalog]: This master's current role is: LEADER
W20260929 03:10:52.435611 28771 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:10:52.435707 28771 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:10:52.435889 28767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:52.436733 28767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:52.442857 28767 catalog_manager.cc:1347] Generated new cluster ID: 332e3845fa7a4cb1b515c2e5c03335a7
I20260929 03:10:52.442960 28767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:52.461159 28767 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:52.462558 28767 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:52.475492 28767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267: Generated new TSK 0
I20260929 03:10:52.476469 28767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab.
Entry for principal oryx/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab.
I20260929 03:10:52.514423 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44895
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab
--principal=oryx/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:52.651901 28777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:52.652176 28777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:52.652225 28777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:52.656299 28777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:52.656363 28777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:52.656404 28777 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:10:52.656437 28777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:52.661744 28777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab
--principal=oryx/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.174.126:44895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:52.662876 28777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:52.663868 28777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:52.673700 28783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:52.673660 28785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:52.673961 28777 server_base.cc:1034] running on GCE node
W20260929 03:10:52.673678 28782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:52.674726 28777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:52.675422 28777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:52.676620 28777 hybrid_clock.cc:648] HybridClock initialized: now 1790651452676596 us; error 29 us; skew 500 ppm
Sep 29 03:10:52 dist-test-slave-k1dg krb5kdc[28660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651452, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:52.679610 28777 init.cc:377] Logged in from keytab as oryx/127.20.174.65@KRBTEST.COM (short username oryx)
I20260929 03:10:52.680526 28777 webserver.cc:460] Webserver started at http://127.20.174.65:35291/ using document root <none> and password file <none>
I20260929 03:10:52.681092 28777 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:52.681180 28777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:52.681407 28777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:52.683528 28777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "7a9824994cc942849265f899aa8c7bbc"
format_stamp: "Formatted at 2026-09-29 03:10:52 on dist-test-slave-k1dg"
server_key: "513c2504dfdffdd434597cd646dd2e8d"
server_key_iv: "a16555af069c146fa400f765274cd7bc"
server_key_version: "encryptionkey@0"
I20260929 03:10:52.684093 28777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "7a9824994cc942849265f899aa8c7bbc"
format_stamp: "Formatted at 2026-09-29 03:10:52 on dist-test-slave-k1dg"
server_key: "513c2504dfdffdd434597cd646dd2e8d"
server_key_iv: "a16555af069c146fa400f765274cd7bc"
server_key_version: "encryptionkey@0"
I20260929 03:10:52.687944 28777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:10:52.690492 28792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:52.691866 28777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:10:52.692056 28777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "7a9824994cc942849265f899aa8c7bbc"
format_stamp: "Formatted at 2026-09-29 03:10:52 on dist-test-slave-k1dg"
server_key: "513c2504dfdffdd434597cd646dd2e8d"
server_key_iv: "a16555af069c146fa400f765274cd7bc"
server_key_version: "encryptionkey@0"
I20260929 03:10:52.692180 28777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:52.703531 28777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:52.706954 28777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:52.707160 28777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:52.708655 28777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:52.708730 28777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:52.708796 28777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:52.708835 28777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:10:52 dist-test-slave-k1dg krb5kdc[28660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651452, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:52.719795 28777 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:44531
I20260929 03:10:52.719871 28907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:44531 every 8 connection(s)
I20260929 03:10:52.720942 28777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:52.722172 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 28777
I20260929 03:10:52.722327 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:52.722720 21177 external_mini_cluster.cc:1428] Setting key 7b160f2ef5f5d7fe1e7356fc6cf704a7
I20260929 03:10:52.724241 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:52.709662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:53571 (local address 127.20.174.126:44895)
0929 03:10:52.709827 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:52.709832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:52.710713 (+ 881us) server_negotiation.cc:407] Connection header received
0929 03:10:52.711857 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:52.711864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:52.711941 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:52.712068 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:52.713449 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.713991 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.716048 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.716467 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.716617 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:52.719766 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:52.719770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:52.719773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:52.719811 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:52.721378 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:52.722347 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:52.722352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:52.722357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:52.722486 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:52.722902 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:52.722906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:52.722907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:52.723195 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:52.723318 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:52.723874 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 03:10:52.724068 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:10:52.725106 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:52.710055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44895 (local address 127.20.174.65:53571)
0929 03:10:52.710540 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:52.710592 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:52.711548 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:52.712316 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:52.712325 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:52.712698 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:52.713293 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:52.713310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.714144 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.714148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:52.715826 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:52.715844 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.716653 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.716657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:52.716855 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:52.717645 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:52.717675 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:52.719561 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:52.721515 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:52.721520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:52.721535 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:52.721879 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:52.722584 (+ 705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:52.722587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:52.722589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:52.722730 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:52.723320 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:52.723330 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:52.723610 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 03:10:52.723811 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 03:10:52.724177 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":140,"threads_started":1}
I20260929 03:10:52.726698 28908 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44895
I20260929 03:10:52.726985 28908 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:52.727586 28908 heartbeater.cc:511] Master 127.20.174.126:44895 requested a full tablet report, sending...
I20260929 03:10:52.729394 28703 ts_manager.cc:194] Registered new tserver with Master: 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531)
I20260929 03:10:52.731206 28703 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:53571
WARNING: no policy specified for oryx/127.20.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.66@KRBTEST.COM" created.
I20260929 03:10:52.736948 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:52.729478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:40241 (local address 127.20.174.126:44895)
0929 03:10:52.729584 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:52.729589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:52.729702 (+ 113us) server_negotiation.cc:407] Connection header received
0929 03:10:52.729849 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:52.729852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:52.729900 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:52.729985 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:52.730418 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.731127 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.732242 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.732474 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.732582 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:52.733712 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:52.733716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:52.733719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:52.733744 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:52.735238 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:52.735666 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:52.735682 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:52.735684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:52.735771 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:52.736131 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:52.736133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:52.736134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:52.736300 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:52.736408 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:52.736661 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 03:10:52.736774 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":45}
I20260929 03:10:52.736974 28915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:52.729149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44895 (local address 127.20.174.65:40241)
0929 03:10:52.729612 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:52.729626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:52.729750 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:52.730000 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:52.730003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:52.730211 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:52.730307 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:52.730312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.731252 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.731258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:52.732126 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:52.732137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.732587 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.732591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:52.732684 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:52.733216 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:52.733232 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:52.733576 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:52.735352 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:52.735357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:52.735360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:52.735567 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:52.735894 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:52.735898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:52.735900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:52.735996 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:52.736396 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:52.736400 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:52.736499 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 03:10:52.736605 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:10:52.736764 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab.
Entry for principal oryx/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab.
I20260929 03:10:52.754400 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44895
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab
--principal=oryx/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:52.870713 28917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:52.870999 28917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:52.871048 28917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:52.875521 28917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:52.875587 28917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:52.875629 28917 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:10:52.875663 28917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:52.881588 28917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab
--principal=oryx/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.174.126:44895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:52.882768 28917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:52.883958 28917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:52.894778 28922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:52.894960 28923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:52.895016 28925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:52.894814 28917 server_base.cc:1034] running on GCE node
I20260929 03:10:52.895674 28917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:52.896493 28917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:52.897702 28917 hybrid_clock.cc:648] HybridClock initialized: now 1790651452897653 us; error 65 us; skew 500 ppm
Sep 29 03:10:52 dist-test-slave-k1dg krb5kdc[28660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651452, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:52.900694 28917 init.cc:377] Logged in from keytab as oryx/127.20.174.66@KRBTEST.COM (short username oryx)
I20260929 03:10:52.901738 28917 webserver.cc:460] Webserver started at http://127.20.174.66:40575/ using document root <none> and password file <none>
I20260929 03:10:52.902323 28917 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:52.902410 28917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:52.902662 28917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:52.904740 28917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "11536f7bfb9f4112a7d275b8e413fb9c"
format_stamp: "Formatted at 2026-09-29 03:10:52 on dist-test-slave-k1dg"
server_key: "aa2fb6ff94b4ceb978981ca641ffc856"
server_key_iv: "0f0055876b193b9e27b4774c2c14f916"
server_key_version: "encryptionkey@0"
I20260929 03:10:52.905304 28917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "11536f7bfb9f4112a7d275b8e413fb9c"
format_stamp: "Formatted at 2026-09-29 03:10:52 on dist-test-slave-k1dg"
server_key: "aa2fb6ff94b4ceb978981ca641ffc856"
server_key_iv: "0f0055876b193b9e27b4774c2c14f916"
server_key_version: "encryptionkey@0"
I20260929 03:10:52.909600 28917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 03:10:52.912696 28932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:52.914110 28917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:10:52.914273 28917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "11536f7bfb9f4112a7d275b8e413fb9c"
format_stamp: "Formatted at 2026-09-29 03:10:52 on dist-test-slave-k1dg"
server_key: "aa2fb6ff94b4ceb978981ca641ffc856"
server_key_iv: "0f0055876b193b9e27b4774c2c14f916"
server_key_version: "encryptionkey@0"
I20260929 03:10:52.914399 28917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:52.931358 28917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:52.934984 28917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:52.935209 28917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:52.936761 28917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:52.936837 28917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:52.936911 28917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:52.936961 28917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:10:52 dist-test-slave-k1dg krb5kdc[28660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651452, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:52.948765 28917 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:35447
I20260929 03:10:52.949204 29047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:35447 every 8 connection(s)
I20260929 03:10:52.949819 28917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:52.951138 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 28917
I20260929 03:10:52.951318 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:52.951683 21177 external_mini_cluster.cc:1428] Setting key 80059cd5be9ee49352b2368c6bd5e27c
I20260929 03:10:52.952780 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:52.937460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:46729 (local address 127.20.174.126:44895)
0929 03:10:52.937613 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:52.937618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:52.938649 (+ 1031us) server_negotiation.cc:407] Connection header received
0929 03:10:52.939744 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:52.939751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:52.939824 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:52.939935 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:52.941452 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.942174 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.943697 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.943916 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.944041 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:52.948214 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:52.948220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:52.948223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:52.948270 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:52.950051 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:52.950807 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:52.950811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:52.950813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:52.950916 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:52.951396 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:52.951403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:52.951405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:52.951658 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:52.951785 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:52.952549 (+ 764us) server_negotiation.cc:299] Negotiation successful
0929 03:10:52.952656 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:10:52.953802 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:52.937964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44895 (local address 127.20.174.66:46729)
0929 03:10:52.938428 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:52.938537 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:52.939444 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:52.940200 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:52.940211 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:52.940605 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:52.941287 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:52.941305 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.942317 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.942321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:52.943559 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:52.943570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.944095 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.944099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:52.944296 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:52.944964 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:52.944984 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:52.948011 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:52.950178 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:52.950184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:52.950200 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:52.950667 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:52.951035 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:52.951038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:52.951041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:52.951247 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:52.951931 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:52.951936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:52.952263 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 03:10:52.952453 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 03:10:52.952831 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":138,"threads_started":1}
I20260929 03:10:52.955134 29048 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44895
I20260929 03:10:52.955294 29048 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:52.956039 29048 heartbeater.cc:511] Master 127.20.174.126:44895 requested a full tablet report, sending...
I20260929 03:10:52.957326 28705 ts_manager.cc:194] Registered new tserver with Master: 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447)
I20260929 03:10:52.958490 28705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:46729
WARNING: no policy specified for oryx/127.20.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.67@KRBTEST.COM" created.
I20260929 03:10:52.966851 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:52.957948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:33831 (local address 127.20.174.126:44895)
0929 03:10:52.958572 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:52.958576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:52.958587 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:10:52.958617 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:52.958619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:52.958654 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:52.958747 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:52.959285 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.960193 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.961248 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.961625 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.961759 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:52.963091 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:52.963094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:52.963096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:52.963119 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:52.964735 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:52.965421 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:52.965427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:52.965431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:52.965527 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:52.965911 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:52.965914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:52.965915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:52.966123 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:52.966203 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:52.966600 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 03:10:52.966700 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546}
I20260929 03:10:52.967191 29055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:52.957656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44895 (local address 127.20.174.66:33831)
0929 03:10:52.957922 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:52.957937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:52.958027 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:52.958764 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:52.958767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:52.959002 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:52.959110 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:52.959141 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.960377 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.960380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:52.961108 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:52.961118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:52.961776 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:52.961780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:52.961858 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:52.962478 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:52.962500 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:52.962864 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:52.965001 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:52.965004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:52.965005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:52.965295 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:52.965633 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:52.965636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:52.965637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:52.965719 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:52.966228 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:52.966234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:52.966350 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 03:10:52.966501 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 03:10:52.966680 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab.
Entry for principal oryx/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab.
I20260929 03:10:52.985575 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44895
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab
--principal=oryx/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:53.105021 29057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:53.105291 29057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:53.105337 29057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:53.109694 29057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:53.109779 29057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:53.109834 29057 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:10:53.109874 29057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:53.115512 29057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab
--principal=oryx/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.174.126:44895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:53.116580 29057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:53.117712 29057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:53.128681 29062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:53.128669 29063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:53.128736 29057 server_base.cc:1034] running on GCE node
W20260929 03:10:53.128634 29065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:53.129310 29057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:53.130014 29057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:53.131201 29057 hybrid_clock.cc:648] HybridClock initialized: now 1790651453131143 us; error 82 us; skew 500 ppm
Sep 29 03:10:53 dist-test-slave-k1dg krb5kdc[28660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651453, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:53.134207 29057 init.cc:377] Logged in from keytab as oryx/127.20.174.67@KRBTEST.COM (short username oryx)
I20260929 03:10:53.135465 29057 webserver.cc:460] Webserver started at http://127.20.174.67:39305/ using document root <none> and password file <none>
I20260929 03:10:53.136174 29057 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:53.136277 29057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:53.136525 29057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:53.138816 29057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "a5e19390aa37412e89ea4f0118263b24"
format_stamp: "Formatted at 2026-09-29 03:10:53 on dist-test-slave-k1dg"
server_key: "120103b0614c77cc0c75208d1e4b2142"
server_key_iv: "b85d7ecc2b71da1f3450f299cfca3bd5"
server_key_version: "encryptionkey@0"
I20260929 03:10:53.139421 29057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "a5e19390aa37412e89ea4f0118263b24"
format_stamp: "Formatted at 2026-09-29 03:10:53 on dist-test-slave-k1dg"
server_key: "120103b0614c77cc0c75208d1e4b2142"
server_key_iv: "b85d7ecc2b71da1f3450f299cfca3bd5"
server_key_version: "encryptionkey@0"
I20260929 03:10:53.143821 29057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:10:53.146737 29072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:53.148167 29057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:10:53.148332 29057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "a5e19390aa37412e89ea4f0118263b24"
format_stamp: "Formatted at 2026-09-29 03:10:53 on dist-test-slave-k1dg"
server_key: "120103b0614c77cc0c75208d1e4b2142"
server_key_iv: "b85d7ecc2b71da1f3450f299cfca3bd5"
server_key_version: "encryptionkey@0"
I20260929 03:10:53.148469 29057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:53.159797 29057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:53.163200 29057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:53.163415 29057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:53.165169 29057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:53.165256 29057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:53.165333 29057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:53.165365 29057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:10:53 dist-test-slave-k1dg krb5kdc[28660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651453, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:53.176450 29057 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:41375
I20260929 03:10:53.176535 29187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:41375 every 8 connection(s)
I20260929 03:10:53.177591 29057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:53.180426 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.165937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:54881 (local address 127.20.174.126:44895)
0929 03:10:53.166106 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.166112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.167488 (+ 1376us) server_negotiation.cc:407] Connection header received
0929 03:10:53.168835 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.168839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.168904 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.169006 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:53.170261 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.170966 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.172496 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.172835 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.172982 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.176191 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:53.176219 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:53.176226 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:53.176263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:53.178248 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:53.178869 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:53.178872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:53.178874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:53.178961 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:53.179304 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:53.179314 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:53.179315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:53.179509 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:53.179590 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.180186 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.180299 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:10:53.181232 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.166843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44895 (local address 127.20.174.67:54881)
0929 03:10:53.167316 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.167370 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.168631 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.169174 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.169181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.169603 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:53.170097 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.170111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.171112 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.171124 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.172346 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.172366 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.173035 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.173081 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.173320 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.173998 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:53.174015 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:53.175996 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:53.178406 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:53.178413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:53.178428 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:53.178758 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:53.179056 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:53.179058 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:53.179060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:53.179197 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:53.179605 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:53.179611 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:53.179850 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.180080 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.180348 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":139,"threads_started":1}
I20260929 03:10:53.182575 29188 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44895
I20260929 03:10:53.182760 29188 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:53.183029 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 29057
I20260929 03:10:53.183172 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:53.183506 29188 heartbeater.cc:511] Master 127.20.174.126:44895 requested a full tablet report, sending...
I20260929 03:10:53.183537 21177 external_mini_cluster.cc:1428] Setting key 382b299a4b665de6265f0aa734610b68
I20260929 03:10:53.184578 28703 ts_manager.cc:194] Registered new tserver with Master: a5e19390aa37412e89ea4f0118263b24 (127.20.174.67:41375)
I20260929 03:10:53.185324 28703 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:54881
I20260929 03:10:53.185958 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:10:53.193666 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.185125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:44069 (local address 127.20.174.126:44895)
0929 03:10:53.185228 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.185232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.185501 (+ 269us) server_negotiation.cc:407] Connection header received
0929 03:10:53.185909 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.185912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.185957 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.186044 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:53.186636 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.187424 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.188571 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.188780 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.188862 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.190151 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:53.190154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:53.190156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:53.190177 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:53.191808 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:53.192294 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:53.192301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:53.192306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:53.192404 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:53.192699 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:53.192706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:53.192710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:53.192960 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:53.193072 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.193437 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.193547 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:10:53.193877 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.185105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44895 (local address 127.20.174.67:44069)
0929 03:10:53.185424 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.185438 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.185566 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.186124 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.186127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.186357 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:53.186507 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.186515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.187569 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.187572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.188433 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.188448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.188901 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.188907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.189017 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.189528 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:53.189547 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:53.190017 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:53.191948 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:53.191954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:53.191955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:53.192165 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:53.192511 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:53.192513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:53.192514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:53.192606 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:53.193112 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:53.193118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:53.193243 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.193378 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.193552 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
Sep 29 03:10:53 dist-test-slave-k1dg krb5kdc[28660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651453, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:10:53 dist-test-slave-k1dg krb5kdc[28660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:53 dist-test-slave-k1dg krb5kdc[28660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:10:53.212827 28754 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:53.205949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.20.174.126:44895)
0929 03:10:53.206175 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.206185 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.206198 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:10:53.206237 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.206239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.206278 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.206385 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:53.207236 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.207813 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.208726 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.208953 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.209072 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.212669 (+ 3597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49550: BlockingRecv error: recv got EOF from 127.0.0.1:49550 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 03:10:53 dist-test-slave-k1dg krb5kdc[28660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651453, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:53.223183 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.214549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.20.174.126:44895)
0929 03:10:53.214666 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.214669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.214835 (+ 166us) server_negotiation.cc:407] Connection header received
0929 03:10:53.214957 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.214960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.215004 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.215074 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:53.215696 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.216200 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.216974 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.217173 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.217247 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.219906 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:53.219909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:53.219912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:53.219939 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:53.221399 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:53.221857 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:53.221860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:53.221861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:53.221944 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:53.222227 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:53.222229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:53.222230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:53.222417 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:53.222693 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.222961 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.223061 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:10:53.226543 28705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:53.229765 28705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:53.245873 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.240033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:41375 (local address 127.0.0.1:36608)
0929 03:10:53.240533 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.240544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.240612 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.241368 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.241370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.241388 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:53.241535 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.241541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.242645 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.242650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.244365 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.244377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.245240 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.245244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.245353 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.245390 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.245507 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.245746 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":291,"thread_start_us":96,"threads_started":1}
I20260929 03:10:53.245975 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.240457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.20.174.67:41375)
0929 03:10:53.240886 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.240897 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.240921 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:10:53.241003 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.241012 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.241218 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.241379 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:53.241647 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.242537 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.244562 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.245105 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.245249 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.245432 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.245711 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.245824 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":107,"threads_started":1}
I20260929 03:10:53.245934 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.240103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.20.174.65:44531)
0929 03:10:53.240849 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.240856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.240879 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:10:53.241004 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.241010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.241223 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.241694 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:53.241831 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.242900 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.244295 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.245160 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.245486 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.245649 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.245705 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.245807 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":80,"threads_started":1}
I20260929 03:10:53.246361 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.239428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:44531 (local address 127.0.0.1:42840)
0929 03:10:53.240278 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.240294 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.240375 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.241378 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.241382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.241400 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:53.241552 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.241559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.243018 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.243025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.244158 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.244168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.245312 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.245316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.245496 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.245521 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.245572 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.245676 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":86,"thread_start_us":80,"threads_started":1}
I20260929 03:10:53.246490 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.239470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.20.174.66:35447)
0929 03:10:53.239912 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.239926 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.240270 (+ 344us) server_negotiation.cc:407] Connection header received
0929 03:10:53.240371 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.240376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.240635 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.241161 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:53.241308 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.242542 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.244508 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.245298 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.245955 (+ 657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.246146 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.246197 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.246283 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":115,"threads_started":1}
I20260929 03:10:53.245873 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.239310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:35447 (local address 127.0.0.1:51750)
0929 03:10:53.239787 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.239845 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.239997 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.240816 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.240822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.240847 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:53.241005 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.241012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.242676 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.242679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.244365 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.244377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.245420 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.245424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.245539 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.245565 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.245618 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.245705 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":191,"thread_start_us":68,"threads_started":1}
I20260929 03:10:53.249279 28842 tablet_service.cc:1467] Processing CreateTablet for tablet 6942027da0844e31ae02d109843cd023 (DEFAULT_TABLE table=test-table [id=b0b571cc5d54460a8d17da9f8392ba44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:53.249331 28982 tablet_service.cc:1467] Processing CreateTablet for tablet 6942027da0844e31ae02d109843cd023 (DEFAULT_TABLE table=test-table [id=b0b571cc5d54460a8d17da9f8392ba44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:53.249308 29122 tablet_service.cc:1467] Processing CreateTablet for tablet 6942027da0844e31ae02d109843cd023 (DEFAULT_TABLE table=test-table [id=b0b571cc5d54460a8d17da9f8392ba44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:53.250334 28842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6942027da0844e31ae02d109843cd023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:53.250349 28982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6942027da0844e31ae02d109843cd023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:53.250349 29122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6942027da0844e31ae02d109843cd023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:53.257138 29216 tablet_bootstrap.cc:492] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc: Bootstrap starting.
I20260929 03:10:53.259594 29216 tablet_bootstrap.cc:654] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:53.260337 29216 log.cc:824] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:53.261077 29218 tablet_bootstrap.cc:492] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: Bootstrap starting.
I20260929 03:10:53.262588 29216 tablet_bootstrap.cc:492] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc: No bootstrap required, opened a new log
I20260929 03:10:53.262799 29216 ts_tablet_manager.cc:1397] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:10:53.263235 29218 tablet_bootstrap.cc:654] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:53.263919 29218 log.cc:824] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:53.264673 29217 tablet_bootstrap.cc:492] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: Bootstrap starting.
I20260929 03:10:53.266103 29216 raft_consensus.cc:348] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.266331 29216 raft_consensus.cc:374] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:53.266394 29216 raft_consensus.cc:729] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9824994cc942849265f899aa8c7bbc, State: Initialized, Role: FOLLOWER
I20260929 03:10:53.266479 29217 tablet_bootstrap.cc:654] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:53.266856 29216 consensus_queue.cc:260] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.267324 29217 log.cc:824] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:53.267513 28908 heartbeater.cc:503] Master 127.20.174.126:44895 was elected leader, sending a full tablet report...
I20260929 03:10:53.267966 29218 tablet_bootstrap.cc:492] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: No bootstrap required, opened a new log
I20260929 03:10:53.268194 29218 ts_tablet_manager.cc:1397] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 03:10:53.269591 29217 tablet_bootstrap.cc:492] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: No bootstrap required, opened a new log
I20260929 03:10:53.269780 29217 ts_tablet_manager.cc:1397] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 03:10:53.272282 29218 raft_consensus.cc:348] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.272583 29218 raft_consensus.cc:374] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:53.272645 29218 raft_consensus.cc:729] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11536f7bfb9f4112a7d275b8e413fb9c, State: Initialized, Role: FOLLOWER
I20260929 03:10:53.273132 29218 consensus_queue.cc:260] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.273715 29217 raft_consensus.cc:348] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.273988 29217 raft_consensus.cc:374] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:53.273981 29048 heartbeater.cc:503] Master 127.20.174.126:44895 was elected leader, sending a full tablet report...
I20260929 03:10:53.274052 29217 raft_consensus.cc:729] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e19390aa37412e89ea4f0118263b24, State: Initialized, Role: FOLLOWER
I20260929 03:10:53.274348 29218 ts_tablet_manager.cc:1428] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:10:53.274520 29217 consensus_queue.cc:260] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.275123 29216 ts_tablet_manager.cc:1428] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20260929 03:10:53.275229 29188 heartbeater.cc:503] Master 127.20.174.126:44895 was elected leader, sending a full tablet report...
I20260929 03:10:53.275501 29217 ts_tablet_manager.cc:1428] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:10:53.371778 29223 raft_consensus.cc:482] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:53.372000 29223 raft_consensus.cc:504] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.373193 29223 leader_election.cc:288] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531), a5e19390aa37412e89ea4f0118263b24 (127.20.174.67:41375)
I20260929 03:10:53.377329 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.373519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:44531 (local address 127.20.174.66:45681)
0929 03:10:53.373653 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.373669 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.373778 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.374126 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.374129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.374147 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:53.374332 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.374339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.375333 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.375337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.376154 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.376163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.376857 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.376861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.376978 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.377006 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.377099 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.377175 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 03:10:53.377312 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.373672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:45681 (local address 127.20.174.65:44531)
0929 03:10:53.373818 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.373823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.373840 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:10:53.373934 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.373939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.374003 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.374099 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:53.374485 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.375211 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.376296 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.376723 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.376873 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.376934 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.377117 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.377187 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:10:53.377736 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.373528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:41375 (local address 127.20.174.66:59679)
0929 03:10:53.374198 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.374210 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.374330 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.374622 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.374625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.374641 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:53.374770 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.374796 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.375516 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.375520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.376616 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.376625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.377292 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.377296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.377406 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.377436 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.377517 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.377583 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":313,"thread_start_us":143,"threads_started":1}
I20260929 03:10:53.377774 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.373717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:59679 (local address 127.20.174.67:41375)
0929 03:10:53.373846 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.373850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.374301 (+ 451us) server_negotiation.cc:407] Connection header received
0929 03:10:53.374477 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.374480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.374520 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.374609 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:53.374897 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.375395 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.376753 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.377169 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.377301 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.377350 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.377564 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.377652 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:10:53.378028 28862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6942027da0844e31ae02d109843cd023" candidate_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9824994cc942849265f899aa8c7bbc" is_pre_election: true
I20260929 03:10:53.378388 28862 raft_consensus.cc:2454] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11536f7bfb9f4112a7d275b8e413fb9c in term 0.
I20260929 03:10:53.378420 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6942027da0844e31ae02d109843cd023" candidate_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e19390aa37412e89ea4f0118263b24" is_pre_election: true
I20260929 03:10:53.378717 29142 raft_consensus.cc:2454] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11536f7bfb9f4112a7d275b8e413fb9c in term 0.
I20260929 03:10:53.378988 28936 leader_election.cc:302] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11536f7bfb9f4112a7d275b8e413fb9c, 7a9824994cc942849265f899aa8c7bbc; no voters:
I20260929 03:10:53.379287 29223 raft_consensus.cc:2781] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:53.379426 29223 raft_consensus.cc:482] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:53.379487 29223 raft_consensus.cc:3037] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:53.380532 29223 raft_consensus.cc:504] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.380897 29223 leader_election.cc:288] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [CANDIDATE]: Term 1 election: Requested vote from peers 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531), a5e19390aa37412e89ea4f0118263b24 (127.20.174.67:41375)
I20260929 03:10:53.381588 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6942027da0844e31ae02d109843cd023" candidate_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e19390aa37412e89ea4f0118263b24"
I20260929 03:10:53.381588 28862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6942027da0844e31ae02d109843cd023" candidate_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9824994cc942849265f899aa8c7bbc"
I20260929 03:10:53.381733 28862 raft_consensus.cc:3037] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:53.381733 29142 raft_consensus.cc:3037] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:53.383298 28862 raft_consensus.cc:2454] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11536f7bfb9f4112a7d275b8e413fb9c in term 1.
I20260929 03:10:53.383697 29142 raft_consensus.cc:2454] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11536f7bfb9f4112a7d275b8e413fb9c in term 1.
I20260929 03:10:53.383901 28936 leader_election.cc:302] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11536f7bfb9f4112a7d275b8e413fb9c, 7a9824994cc942849265f899aa8c7bbc; no voters:
I20260929 03:10:53.384214 29223 raft_consensus.cc:2781] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:53.384487 29223 raft_consensus.cc:686] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 LEADER]: Becoming Leader. State: Replica: 11536f7bfb9f4112a7d275b8e413fb9c, State: Running, Role: LEADER
I20260929 03:10:53.384860 29223 consensus_queue.cc:237] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.388217 28705 catalog_manager.cc:5495] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66). New cstate: current_term: 1 leader_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } health_report { overall_health: UNKNOWN } } }
W20260929 03:10:53.429119 29189 tablet.cc:2369] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:10:53.437784 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.433867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.20.174.66:35447)
0929 03:10:53.433998 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.434002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.434016 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:10:53.434156 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.434159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.434215 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.434308 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:53.434617 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.435122 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.436079 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.436363 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.436467 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.436765 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:53.437286 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:53.437403 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.437598 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.437666 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:10:53.444772 28862 raft_consensus.cc:1261] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Refusing update from remote peer 11536f7bfb9f4112a7d275b8e413fb9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:53.444772 29142 raft_consensus.cc:1261] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Refusing update from remote peer 11536f7bfb9f4112a7d275b8e413fb9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:53.445719 29223 consensus_queue.cc:1035] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:53.446022 29226 consensus_queue.cc:1035] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:10:53.451710 29049 tablet.cc:2369] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:10:53.454545 29229 mvcc.cc:205] Tried to move back new op lower bound from 7334508353299759104 to 7334508353067192320. Current Snapshot: MvccSnapshot[applied={T|T < 7334508353299759104}]
I20260929 03:10:53.454883 29230 mvcc.cc:205] Tried to move back new op lower bound from 7334508353299759104 to 7334508353067192320. Current Snapshot: MvccSnapshot[applied={T|T < 7334508353299759104}]
I20260929 03:10:53.464707 28705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:53.464915 28705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:53.467614 28705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267: Sending DeleteTablet for 3 replicas of tablet 6942027da0844e31ae02d109843cd023
I20260929 03:10:53.468533 29122 tablet_service.cc:1514] Processing DeleteTablet for tablet 6942027da0844e31ae02d109843cd023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:53 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:36608
I20260929 03:10:53.468530 28842 tablet_service.cc:1514] Processing DeleteTablet for tablet 6942027da0844e31ae02d109843cd023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:53 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:42840
I20260929 03:10:53.469079 29238 tablet_replica.cc:330] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc: stopping tablet replica
I20260929 03:10:53.469056 28982 tablet_service.cc:1514] Processing DeleteTablet for tablet 6942027da0844e31ae02d109843cd023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:53 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:51750
I20260929 03:10:53.469461 29238 raft_consensus.cc:2229] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:53.469524 29240 tablet_replica.cc:330] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: stopping tablet replica
I20260929 03:10:53.469836 29238 raft_consensus.cc:2258] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:53.469983 29240 raft_consensus.cc:2229] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:53.470418 28705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:53.470923 28705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:53.471148 29240 raft_consensus.cc:2258] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:53.471608 29238 ts_tablet_manager.cc:1905] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:53.472836 29240 ts_tablet_manager.cc:1905] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:53.474659 29239 tablet_replica.cc:330] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: stopping tablet replica
I20260929 03:10:53.475008 29239 raft_consensus.cc:2229] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:53.475337 29239 raft_consensus.cc:2258] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:53.475888 29238 ts_tablet_manager.cc:1918] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:53.476013 29238 log.cc:1196] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/wal/wals/6942027da0844e31ae02d109843cd023
I20260929 03:10:53.476246 29240 ts_tablet_manager.cc:1918] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:53.476336 29238 ts_tablet_manager.cc:1939] T 6942027da0844e31ae02d109843cd023 P 7a9824994cc942849265f899aa8c7bbc: Deleting consensus metadata
I20260929 03:10:53.476553 29240 log.cc:1196] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/6942027da0844e31ae02d109843cd023
I20260929 03:10:53.476961 29240 ts_tablet_manager.cc:1939] T 6942027da0844e31ae02d109843cd023 P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting consensus metadata
I20260929 03:10:53.478109 29122 tablet_service.cc:1467] Processing CreateTablet for tablet 511e95cb60a44addad9b731c7cc9148e (DEFAULT_TABLE table=test-table [id=e774b47491d14843b0e7d100af06cd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:53.478166 28982 tablet_service.cc:1467] Processing CreateTablet for tablet 511e95cb60a44addad9b731c7cc9148e (DEFAULT_TABLE table=test-table [id=e774b47491d14843b0e7d100af06cd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:53.478395 29122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 511e95cb60a44addad9b731c7cc9148e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:53.478478 28982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 511e95cb60a44addad9b731c7cc9148e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:53.480937 29218 tablet_bootstrap.cc:492] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c: Bootstrap starting.
I20260929 03:10:53.481092 28689 catalog_manager.cc:4841] TS 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447): tablet 6942027da0844e31ae02d109843cd023 (table test-table [id=b0b571cc5d54460a8d17da9f8392ba44]) successfully deleted
I20260929 03:10:53.481552 28692 catalog_manager.cc:4841] TS 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531): tablet 6942027da0844e31ae02d109843cd023 (table test-table [id=b0b571cc5d54460a8d17da9f8392ba44]) successfully deleted
I20260929 03:10:53.481974 29218 tablet_bootstrap.cc:654] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:53.482120 28842 tablet_service.cc:1467] Processing CreateTablet for tablet 511e95cb60a44addad9b731c7cc9148e (DEFAULT_TABLE table=test-table [id=e774b47491d14843b0e7d100af06cd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:53.482455 28842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 511e95cb60a44addad9b731c7cc9148e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:53.482703 29217 tablet_bootstrap.cc:492] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24: Bootstrap starting.
I20260929 03:10:53.483712 29217 tablet_bootstrap.cc:654] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:53.484941 29218 tablet_bootstrap.cc:492] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c: No bootstrap required, opened a new log
I20260929 03:10:53.485013 29216 tablet_bootstrap.cc:492] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc: Bootstrap starting.
I20260929 03:10:53.485029 29218 ts_tablet_manager.cc:1397] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:10:53.485503 29218 raft_consensus.cc:348] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.485615 29218 raft_consensus.cc:374] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:53.485643 29218 raft_consensus.cc:729] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11536f7bfb9f4112a7d275b8e413fb9c, State: Initialized, Role: FOLLOWER
I20260929 03:10:53.485762 29218 consensus_queue.cc:260] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.486008 29218 ts_tablet_manager.cc:1428] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:53.486020 29216 tablet_bootstrap.cc:654] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:53.486655 29217 tablet_bootstrap.cc:492] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24: No bootstrap required, opened a new log
I20260929 03:10:53.486743 29217 ts_tablet_manager.cc:1397] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:10:53.487180 29217 raft_consensus.cc:348] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.487296 29217 raft_consensus.cc:374] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:53.487327 29217 raft_consensus.cc:729] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e19390aa37412e89ea4f0118263b24, State: Initialized, Role: FOLLOWER
I20260929 03:10:53.487468 29217 consensus_queue.cc:260] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.488287 29217 ts_tablet_manager.cc:1428] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:53.490801 29239 ts_tablet_manager.cc:1905] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:53.493392 29239 ts_tablet_manager.cc:1918] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:53.493516 29239 log.cc:1196] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/6942027da0844e31ae02d109843cd023
I20260929 03:10:53.493860 29239 ts_tablet_manager.cc:1939] T 6942027da0844e31ae02d109843cd023 P a5e19390aa37412e89ea4f0118263b24: Deleting consensus metadata
I20260929 03:10:53.494094 29216 tablet_bootstrap.cc:492] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc: No bootstrap required, opened a new log
I20260929 03:10:53.494196 29216 ts_tablet_manager.cc:1397] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260929 03:10:53.494874 29216 raft_consensus.cc:348] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.495012 29216 raft_consensus.cc:374] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:53.495056 29216 raft_consensus.cc:729] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9824994cc942849265f899aa8c7bbc, State: Initialized, Role: FOLLOWER
I20260929 03:10:53.495225 29216 consensus_queue.cc:260] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.495360 28689 catalog_manager.cc:4841] TS a5e19390aa37412e89ea4f0118263b24 (127.20.174.67:41375): tablet 6942027da0844e31ae02d109843cd023 (table test-table [id=b0b571cc5d54460a8d17da9f8392ba44]) successfully deleted
I20260929 03:10:53.495999 29216 ts_tablet_manager.cc:1428] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260929 03:10:53.722715 28909 tablet.cc:2369] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:10:53.729646 29224 raft_consensus.cc:482] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:53.729815 29224 raft_consensus.cc:504] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.731177 29224 leader_election.cc:288] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531), 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447)
I20260929 03:10:53.736236 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.731721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:60825 (local address 127.20.174.66:35447)
0929 03:10:53.731835 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.731840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.731859 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:10:53.732141 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.732145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.732197 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.732292 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:53.732688 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.733456 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.735073 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.735593 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.735725 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.735791 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.736013 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.736097 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:10:53.736254 29244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.731492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:35447 (local address 127.20.174.67:60825)
0929 03:10:53.731771 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.731789 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.731907 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.732365 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.732369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.732398 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:53.732541 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.732549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.733569 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.733572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.734930 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.734940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.735748 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.735752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.735865 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.735893 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.735989 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.736078 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
I20260929 03:10:53.736503 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.731965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:46293 (local address 127.20.174.65:44531)
0929 03:10:53.732072 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.732076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.732866 (+ 790us) server_negotiation.cc:407] Connection header received
0929 03:10:53.732959 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.732962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.733008 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.733105 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:53.733366 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.734005 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.735140 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.735752 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.735868 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.735929 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.736257 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.736344 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:10:53.736912 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.731492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:44531 (local address 127.20.174.67:46293)
0929 03:10:53.732761 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.732775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.732867 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.733129 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.733131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.733146 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:53.733251 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.733257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.734139 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.734143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.735025 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.735037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.735994 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.735997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.736108 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.736131 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.736705 (+ 574us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.736794 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"mutex_wait_us":878,"thread_start_us":98,"threads_started":1}
I20260929 03:10:53.736847 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511e95cb60a44addad9b731c7cc9148e" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" is_pre_election: true
I20260929 03:10:53.737115 29002 raft_consensus.cc:2454] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 0.
I20260929 03:10:53.737408 28862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511e95cb60a44addad9b731c7cc9148e" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9824994cc942849265f899aa8c7bbc" is_pre_election: true
I20260929 03:10:53.737567 28862 raft_consensus.cc:2454] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 0.
I20260929 03:10:53.737610 29073 leader_election.cc:302] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11536f7bfb9f4112a7d275b8e413fb9c, a5e19390aa37412e89ea4f0118263b24; no voters:
I20260929 03:10:53.737897 29224 raft_consensus.cc:2781] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:53.737982 29224 raft_consensus.cc:482] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:53.738039 29224 raft_consensus.cc:3037] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:53.739125 29224 raft_consensus.cc:504] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.739512 29224 leader_election.cc:288] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 election: Requested vote from peers 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531), 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447)
I20260929 03:10:53.739888 28862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511e95cb60a44addad9b731c7cc9148e" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9824994cc942849265f899aa8c7bbc"
I20260929 03:10:53.740020 28862 raft_consensus.cc:3037] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:53.740142 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511e95cb60a44addad9b731c7cc9148e" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11536f7bfb9f4112a7d275b8e413fb9c"
I20260929 03:10:53.740280 29002 raft_consensus.cc:3037] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:53.741022 28862 raft_consensus.cc:2454] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 1.
I20260929 03:10:53.741331 29076 leader_election.cc:302] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9824994cc942849265f899aa8c7bbc, a5e19390aa37412e89ea4f0118263b24; no voters:
I20260929 03:10:53.741456 29002 raft_consensus.cc:2454] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 1.
I20260929 03:10:53.741501 29224 raft_consensus.cc:2781] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:53.741685 29224 raft_consensus.cc:686] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 1 LEADER]: Becoming Leader. State: Replica: a5e19390aa37412e89ea4f0118263b24, State: Running, Role: LEADER
I20260929 03:10:53.742015 29224 consensus_queue.cc:237] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.744580 28705 catalog_manager.cc:5495] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e19390aa37412e89ea4f0118263b24 (127.20.174.67). New cstate: current_term: 1 leader_uuid: "a5e19390aa37412e89ea4f0118263b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } health_report { overall_health: HEALTHY } } }
Sep 29 03:10:53 dist-test-slave-k1dg krb5kdc[28660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651452, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.126@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:53.813977 29254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.804412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.20.174.126:44895)
0929 03:10:53.804669 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.804676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.804691 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:53.804848 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.804851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.804891 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.805010 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:53.805644 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.806201 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.807139 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.807357 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.807439 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.810109 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:53.810113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:53.810115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:53.810140 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:53.811937 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:53.812561 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:53.812565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:53.812567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:53.812660 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:53.812963 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:53.812966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:53.812967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:53.813199 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:53.813306 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.813630 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.813839 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":197,"thread_start_us":106,"threads_started":1}
I20260929 03:10:53.813977 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.804249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44895 (local address 127.0.0.1:49560)
0929 03:10:53.804566 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.804582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.804696 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.805030 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.805032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.805425 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:53.805531 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.805536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.806339 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.806342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.807015 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.807026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.807483 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.807485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.807549 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.808316 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:53.808329 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:53.809941 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:53.812092 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:53.812103 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:53.812108 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:53.812368 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:53.812774 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:53.812779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:53.812787 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:53.812862 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:53.813314 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:53.813324 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:53.813442 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.813570 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.813769 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":99,"threads_started":1}
I20260929 03:10:53.818387 28705 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:49560:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:10:53.819223 28705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:53.824591 28842 tablet_service.cc:1467] Processing CreateTablet for tablet eae534208ca5479daec72a28b07dd9fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b06b4ec367334589bc888b6ca8c76609]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:10:53.824716 28982 tablet_service.cc:1467] Processing CreateTablet for tablet eae534208ca5479daec72a28b07dd9fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b06b4ec367334589bc888b6ca8c76609]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:10:53.824808 29122 tablet_service.cc:1467] Processing CreateTablet for tablet eae534208ca5479daec72a28b07dd9fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b06b4ec367334589bc888b6ca8c76609]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:10:53.825012 28842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae534208ca5479daec72a28b07dd9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:53.825136 29122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae534208ca5479daec72a28b07dd9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:53.825136 28982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae534208ca5479daec72a28b07dd9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:53.828148 29216 tablet_bootstrap.cc:492] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc: Bootstrap starting.
I20260929 03:10:53.828248 29218 tablet_bootstrap.cc:492] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c: Bootstrap starting.
I20260929 03:10:53.828346 29217 tablet_bootstrap.cc:492] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: Bootstrap starting.
I20260929 03:10:53.829157 29216 tablet_bootstrap.cc:654] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:53.829283 29218 tablet_bootstrap.cc:654] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:53.829300 29217 tablet_bootstrap.cc:654] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:53.830633 29218 tablet_bootstrap.cc:492] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c: No bootstrap required, opened a new log
I20260929 03:10:53.830724 29218 ts_tablet_manager.cc:1397] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:53.830904 29216 tablet_bootstrap.cc:492] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc: No bootstrap required, opened a new log
I20260929 03:10:53.830994 29216 ts_tablet_manager.cc:1397] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 03:10:53.830998 29217 tablet_bootstrap.cc:492] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: No bootstrap required, opened a new log
I20260929 03:10:53.831087 29217 ts_tablet_manager.cc:1397] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:53.831187 29218 raft_consensus.cc:348] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.831328 29218 raft_consensus.cc:374] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:53.831369 29218 raft_consensus.cc:729] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11536f7bfb9f4112a7d275b8e413fb9c, State: Initialized, Role: FOLLOWER
I20260929 03:10:53.831444 29216 raft_consensus.cc:348] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.831558 29216 raft_consensus.cc:374] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:53.831521 29218 consensus_queue.cc:260] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.831595 29216 raft_consensus.cc:729] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9824994cc942849265f899aa8c7bbc, State: Initialized, Role: FOLLOWER
I20260929 03:10:53.831612 29217 raft_consensus.cc:348] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.831718 29216 consensus_queue.cc:260] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.831789 29217 raft_consensus.cc:374] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:53.831832 29217 raft_consensus.cc:729] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e19390aa37412e89ea4f0118263b24, State: Initialized, Role: FOLLOWER
I20260929 03:10:53.831998 29216 ts_tablet_manager.cc:1428] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:53.831960 29217 consensus_queue.cc:260] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.832085 29223 tablet_replica.cc:439] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } } }
I20260929 03:10:53.832226 29223 tablet_replica.cc:442] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:53.832244 29217 ts_tablet_manager.cc:1428] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:53.832320 29221 tablet_replica.cc:439] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } } }
I20260929 03:10:53.832481 29221 tablet_replica.cc:442] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:53.832424 29224 tablet_replica.cc:439] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } } }
I20260929 03:10:53.832547 29224 tablet_replica.cc:442] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:53.833678 29218 ts_tablet_manager.cc:1428] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:53.904995 29224 raft_consensus.cc:482] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:53.905140 29224 raft_consensus.cc:504] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.905680 29224 leader_election.cc:288] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531), 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447)
I20260929 03:10:53.906165 28862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae534208ca5479daec72a28b07dd9fd" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9824994cc942849265f899aa8c7bbc" is_pre_election: true
I20260929 03:10:53.906258 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae534208ca5479daec72a28b07dd9fd" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" is_pre_election: true
I20260929 03:10:53.906347 28862 raft_consensus.cc:2454] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 0.
I20260929 03:10:53.906435 29002 raft_consensus.cc:2454] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 0.
I20260929 03:10:53.906711 29076 leader_election.cc:302] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9824994cc942849265f899aa8c7bbc, a5e19390aa37412e89ea4f0118263b24; no voters:
I20260929 03:10:53.906920 29224 raft_consensus.cc:2781] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:53.906993 29224 raft_consensus.cc:482] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:53.907073 29224 raft_consensus.cc:3037] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:53.908094 29224 raft_consensus.cc:504] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.908444 29224 leader_election.cc:288] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 election: Requested vote from peers 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531), 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447)
I20260929 03:10:53.908806 28862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae534208ca5479daec72a28b07dd9fd" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9824994cc942849265f899aa8c7bbc"
I20260929 03:10:53.908820 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae534208ca5479daec72a28b07dd9fd" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11536f7bfb9f4112a7d275b8e413fb9c"
I20260929 03:10:53.908947 28862 raft_consensus.cc:3037] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:53.908962 29002 raft_consensus.cc:3037] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:53.909919 29002 raft_consensus.cc:2454] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 1.
I20260929 03:10:53.909924 28862 raft_consensus.cc:2454] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 1.
I20260929 03:10:53.910238 29073 leader_election.cc:302] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11536f7bfb9f4112a7d275b8e413fb9c, a5e19390aa37412e89ea4f0118263b24; no voters:
I20260929 03:10:53.910424 29224 raft_consensus.cc:2781] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:53.910573 29224 raft_consensus.cc:686] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [term 1 LEADER]: Becoming Leader. State: Replica: a5e19390aa37412e89ea4f0118263b24, State: Running, Role: LEADER
I20260929 03:10:53.910712 29224 consensus_queue.cc:237] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:53.911427 29224 tablet_replica.cc:439] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: TxnStatusTablet state changed. Reason: New leader a5e19390aa37412e89ea4f0118263b24. Latest consensus state: current_term: 1 leader_uuid: "a5e19390aa37412e89ea4f0118263b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } } }
I20260929 03:10:53.911540 29224 tablet_replica.cc:442] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:10:53.911845 29259 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:10:53.912034 29259 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:10:53.912803 28700 catalog_manager.cc:5495] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e19390aa37412e89ea4f0118263b24 (127.20.174.67). New cstate: current_term: 1 leader_uuid: "a5e19390aa37412e89ea4f0118263b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } health_report { overall_health: HEALTHY } } }
I20260929 03:10:53.936656 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.932194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:41375 (local address 127.0.0.1:36618)
0929 03:10:53.932364 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.932382 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.932486 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.933227 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.933231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.933239 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:10:53.933354 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.933359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.934162 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.934166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.934927 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.934937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.935440 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.935451 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.935542 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.935593 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:10:53.936314 (+ 721us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:10:53.936347 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.936424 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.936524 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 03:10:53.936672 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.932649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.20.174.67:41375)
0929 03:10:53.932933 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.932940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.932958 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:10:53.933024 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.933027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.933076 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.933207 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:53.933482 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.934037 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.935060 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.935322 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.935437 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.935772 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:53.936213 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:53.936346 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.936448 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.936506 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":118,"threads_started":1}
I20260929 03:10:53.939715 28862 raft_consensus.cc:1261] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Refusing update from remote peer a5e19390aa37412e89ea4f0118263b24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:53.939749 29002 raft_consensus.cc:1261] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Refusing update from remote peer a5e19390aa37412e89ea4f0118263b24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:53.940264 29224 consensus_queue.cc:1035] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:53.940577 29246 consensus_queue.cc:1035] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:53.942724 29223 tablet_replica.cc:439] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c: TxnStatusTablet state changed. Reason: New leader a5e19390aa37412e89ea4f0118263b24. Latest consensus state: current_term: 1 leader_uuid: "a5e19390aa37412e89ea4f0118263b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } } }
I20260929 03:10:53.942878 29223 tablet_replica.cc:442] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:53.943312 29221 tablet_replica.cc:439] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc: TxnStatusTablet state changed. Reason: New leader a5e19390aa37412e89ea4f0118263b24. Latest consensus state: current_term: 1 leader_uuid: "a5e19390aa37412e89ea4f0118263b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } } }
I20260929 03:10:53.943436 29221 tablet_replica.cc:442] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:53.944823 29246 tablet_replica.cc:439] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5e19390aa37412e89ea4f0118263b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } } }
I20260929 03:10:53.944965 29246 tablet_replica.cc:442] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:10:53.945072 29232 mvcc.cc:205] Tried to move back new op lower bound from 7334508355331125248 to 7334508355219906560. Current Snapshot: MvccSnapshot[applied={T|T < 7334508355331125248}]
I20260929 03:10:53.946115 29221 tablet_replica.cc:439] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5e19390aa37412e89ea4f0118263b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } } }
I20260929 03:10:53.946247 29221 tablet_replica.cc:442] T eae534208ca5479daec72a28b07dd9fd P 7a9824994cc942849265f899aa8c7bbc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:53.948436 29224 tablet_replica.cc:439] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5e19390aa37412e89ea4f0118263b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } } }
I20260929 03:10:53.948585 29224 tablet_replica.cc:442] T eae534208ca5479daec72a28b07dd9fd P a5e19390aa37412e89ea4f0118263b24: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:10:53.949704 29223 tablet_replica.cc:439] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5e19390aa37412e89ea4f0118263b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } } }
I20260929 03:10:53.949829 29223 tablet_replica.cc:442] T eae534208ca5479daec72a28b07dd9fd P 11536f7bfb9f4112a7d275b8e413fb9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:10:53.955323 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.951573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.20.174.67:41375)
0929 03:10:53.951703 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.951707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.951720 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:10:53.951803 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.951807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.951853 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.951943 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:53.952235 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.952901 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.953950 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.954186 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.954304 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.954629 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:53.954712 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:53.954756 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.955092 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.955144 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:10:53.967087 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.962894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:46151 (local address 127.20.174.67:41375)
0929 03:10:53.962998 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:53.963003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:53.963271 (+ 268us) server_negotiation.cc:407] Connection header received
0929 03:10:53.963380 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:53.963384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:53.963431 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:53.963506 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:53.963788 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.964550 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.965822 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.966166 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.966278 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.966618 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:53.966700 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:53.966771 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:53.966867 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:10:53.966918 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:10:53.967125 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:53.962821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:41375 (local address 127.20.174.67:46151)
0929 03:10:53.963164 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:53.963176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:53.963288 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:53.963535 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:53.963538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:53.963545 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:10:53.963682 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.963687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.964658 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.964661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.965693 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:53.965704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:53.966283 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:53.966286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:53.966398 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:53.966476 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:10:53.966787 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:10:53.966794 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:10:53.966859 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:10:53.966941 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":88,"threads_started":1}
I20260929 03:10:53.974011 29002 raft_consensus.cc:1261] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Refusing update from remote peer a5e19390aa37412e89ea4f0118263b24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:53.974126 28862 raft_consensus.cc:1261] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Refusing update from remote peer a5e19390aa37412e89ea4f0118263b24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:53.974573 29224 consensus_queue.cc:1035] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:53.974843 29265 consensus_queue.cc:1035] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:54.019719 28700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:54.019915 28700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:54.022168 28700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267: Sending DeleteTablet for 3 replicas of tablet 511e95cb60a44addad9b731c7cc9148e
I20260929 03:10:54.022830 28842 tablet_service.cc:1514] Processing DeleteTablet for tablet 511e95cb60a44addad9b731c7cc9148e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:54 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:42840
I20260929 03:10:54.023039 28982 tablet_service.cc:1514] Processing DeleteTablet for tablet 511e95cb60a44addad9b731c7cc9148e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:54 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:51750
I20260929 03:10:54.023195 29274 tablet_replica.cc:330] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc: stopping tablet replica
I20260929 03:10:54.023321 29274 raft_consensus.cc:2229] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:54.023448 29274 raft_consensus.cc:2258] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:54.023528 29122 tablet_service.cc:1514] Processing DeleteTablet for tablet 511e95cb60a44addad9b731c7cc9148e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:54 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:36608
I20260929 03:10:54.023840 29275 tablet_replica.cc:330] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c: stopping tablet replica
I20260929 03:10:54.023983 29275 raft_consensus.cc:2229] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:54.024107 29275 raft_consensus.cc:2258] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:54.024340 29276 tablet_replica.cc:330] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24: stopping tablet replica
I20260929 03:10:54.024474 29276 raft_consensus.cc:2229] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:54.024730 29274 ts_tablet_manager.cc:1905] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:54.024801 29276 raft_consensus.cc:2258] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:54.025319 29275 ts_tablet_manager.cc:1905] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:54.025522 29276 ts_tablet_manager.cc:1905] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:54.028090 29274 ts_tablet_manager.cc:1918] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:54.028182 29274 log.cc:1196] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/wal/wals/511e95cb60a44addad9b731c7cc9148e
I20260929 03:10:54.028460 29274 ts_tablet_manager.cc:1939] T 511e95cb60a44addad9b731c7cc9148e P 7a9824994cc942849265f899aa8c7bbc: Deleting consensus metadata
I20260929 03:10:54.028891 29276 ts_tablet_manager.cc:1918] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:54.029048 29276 log.cc:1196] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/511e95cb60a44addad9b731c7cc9148e
I20260929 03:10:54.029242 29275 ts_tablet_manager.cc:1918] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:54.029325 29276 ts_tablet_manager.cc:1939] T 511e95cb60a44addad9b731c7cc9148e P a5e19390aa37412e89ea4f0118263b24: Deleting consensus metadata
I20260929 03:10:54.029373 29275 log.cc:1196] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/511e95cb60a44addad9b731c7cc9148e
I20260929 03:10:54.029645 29275 ts_tablet_manager.cc:1939] T 511e95cb60a44addad9b731c7cc9148e P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting consensus metadata
I20260929 03:10:54.030648 28689 catalog_manager.cc:4841] TS 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447): tablet 511e95cb60a44addad9b731c7cc9148e (table test-table [id=e774b47491d14843b0e7d100af06cd5e]) successfully deleted
I20260929 03:10:54.030961 28692 catalog_manager.cc:4841] TS 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531): tablet 511e95cb60a44addad9b731c7cc9148e (table test-table [id=e774b47491d14843b0e7d100af06cd5e]) successfully deleted
I20260929 03:10:54.031026 28689 catalog_manager.cc:4841] TS a5e19390aa37412e89ea4f0118263b24 (127.20.174.67:41375): tablet 511e95cb60a44addad9b731c7cc9148e (table test-table [id=e774b47491d14843b0e7d100af06cd5e]) successfully deleted
Sep 29 03:10:54 dist-test-slave-k1dg krb5kdc[28660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:10:54 dist-test-slave-k1dg krb5kdc[28660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:54 dist-test-slave-k1dg krb5kdc[28660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:10:54.057461 29254 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:54.049367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.20.174.126:44895)
0929 03:10:54.049519 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:54.049527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:54.049734 (+ 207us) server_negotiation.cc:407] Connection header received
0929 03:10:54.049865 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:54.049868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:54.049921 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:54.050003 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:54.051048 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.051685 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.052983 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.053268 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.053403 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:54.057322 (+ 3919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49566: BlockingRecv error: recv got EOF from 127.0.0.1:49566 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 03:10:54 dist-test-slave-k1dg krb5kdc[28660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM
I20260929 03:10:54.069934 29254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:54.059513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49574 (local address 127.20.174.126:44895)
0929 03:10:54.059647 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:54.059659 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:54.059752 (+ 93us) server_negotiation.cc:407] Connection header received
0929 03:10:54.060053 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:54.060057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:54.060118 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:54.060201 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:54.061027 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.061560 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.062793 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.063228 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.063399 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:54.066341 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:54.066345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:54.066348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:54.066387 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:54.068086 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:54.068570 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:54.068574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:54.068576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:54.068665 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:54.068971 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:54.068974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:54.068976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:54.069173 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:54.069324 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:54.069612 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 03:10:54.069754 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:10:54.073060 28700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:54.073560 28700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:54.079234 28982 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3c710b333b45eb9b62bef317536b8e (DEFAULT_TABLE table=test-table [id=202c82bb912c4f8f88556cd739544bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:54.079327 28842 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3c710b333b45eb9b62bef317536b8e (DEFAULT_TABLE table=test-table [id=202c82bb912c4f8f88556cd739544bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:54.079810 28982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3c710b333b45eb9b62bef317536b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:54.079900 28842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3c710b333b45eb9b62bef317536b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:54.081812 29218 tablet_bootstrap.cc:492] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c: Bootstrap starting.
I20260929 03:10:54.082696 29122 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3c710b333b45eb9b62bef317536b8e (DEFAULT_TABLE table=test-table [id=202c82bb912c4f8f88556cd739544bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:54.082782 29218 tablet_bootstrap.cc:654] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:54.083011 29122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3c710b333b45eb9b62bef317536b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:54.083982 29218 tablet_bootstrap.cc:492] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c: No bootstrap required, opened a new log
I20260929 03:10:54.084095 29218 ts_tablet_manager.cc:1397] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:10:54.084542 29218 raft_consensus.cc:348] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.084672 29218 raft_consensus.cc:374] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:54.084713 29218 raft_consensus.cc:729] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11536f7bfb9f4112a7d275b8e413fb9c, State: Initialized, Role: FOLLOWER
I20260929 03:10:54.084839 29218 consensus_queue.cc:260] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.085143 29218 ts_tablet_manager.cc:1428] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:54.086647 29217 tablet_bootstrap.cc:492] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24: Bootstrap starting.
I20260929 03:10:54.086776 29216 tablet_bootstrap.cc:492] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc: Bootstrap starting.
I20260929 03:10:54.087690 29217 tablet_bootstrap.cc:654] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:54.087819 29216 tablet_bootstrap.cc:654] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:54.089170 29216 tablet_bootstrap.cc:492] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc: No bootstrap required, opened a new log
I20260929 03:10:54.089294 29216 ts_tablet_manager.cc:1397] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 03:10:54.089946 29216 raft_consensus.cc:348] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.090072 29216 raft_consensus.cc:374] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:54.090081 29217 tablet_bootstrap.cc:492] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24: No bootstrap required, opened a new log
I20260929 03:10:54.090111 29216 raft_consensus.cc:729] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9824994cc942849265f899aa8c7bbc, State: Initialized, Role: FOLLOWER
I20260929 03:10:54.090162 29217 ts_tablet_manager.cc:1397] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:10:54.090241 29216 consensus_queue.cc:260] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.090658 29217 raft_consensus.cc:348] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.090785 29217 raft_consensus.cc:374] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:54.090818 29217 raft_consensus.cc:729] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e19390aa37412e89ea4f0118263b24, State: Initialized, Role: FOLLOWER
I20260929 03:10:54.090947 29217 consensus_queue.cc:260] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.091233 29217 ts_tablet_manager.cc:1428] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:10:54.091921 29216 ts_tablet_manager.cc:1428] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 03:10:54.193413 29221 raft_consensus.cc:482] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:54.193579 29221 raft_consensus.cc:504] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.194876 29221 leader_election.cc:288] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447), a5e19390aa37412e89ea4f0118263b24 (127.20.174.67:41375)
I20260929 03:10:54.201195 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:54.195199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:54937 (local address 127.20.174.66:35447)
0929 03:10:54.195346 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:54.195353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:54.195948 (+ 595us) server_negotiation.cc:407] Connection header received
0929 03:10:54.196175 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:54.196178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:54.196224 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:54.196276 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:54.196880 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.197653 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.199684 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.200303 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.200401 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:54.200455 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:54.200968 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 03:10:54.201056 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:10:54.201195 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:54.195608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:36893 (local address 127.20.174.67:41375)
0929 03:10:54.195735 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:54.195740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:54.196011 (+ 271us) server_negotiation.cc:407] Connection header received
0929 03:10:54.196174 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:54.196177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:54.196223 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:54.196282 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:10:54.196855 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.197653 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.199683 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.200311 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.200401 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:54.200455 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:54.200968 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 03:10:54.201056 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:10:54.201634 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:54.195088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:35447 (local address 127.20.174.65:54937)
0929 03:10:54.195848 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:54.195862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:54.196038 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:54.196423 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:54.196426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:54.196466 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:54.196698 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:54.196705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.197842 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.197848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:54.199526 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:54.199538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.200557 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.200561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:54.200794 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:54.200843 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 03:10:54.201349 (+ 506us) client_negotiation.cc:240] Negotiation successful
0929 03:10:54.201455 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":252,"spinlock_wait_cycles":3840,"thread_start_us":72,"threads_started":1}
I20260929 03:10:54.202111 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3c710b333b45eb9b62bef317536b8e" candidate_uuid: "7a9824994cc942849265f899aa8c7bbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" is_pre_election: true
I20260929 03:10:54.202279 29002 raft_consensus.cc:2454] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9824994cc942849265f899aa8c7bbc in term 0.
I20260929 03:10:54.202625 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:54.195472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:41375 (local address 127.20.174.65:36893)
0929 03:10:54.195897 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:54.195911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:54.196058 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:54.196423 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:54.196426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:54.196465 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:10:54.196697 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:54.196705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.197822 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.197827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:54.199526 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:54.199538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.200557 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.200561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:54.200799 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:54.200842 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:10:54.202110 (+ 1268us) client_negotiation.cc:240] Negotiation successful
0929 03:10:54.202193 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":374,"spinlock_wait_cycles":29312,"thread_start_us":101,"threads_started":1}
I20260929 03:10:54.202636 28793 leader_election.cc:302] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11536f7bfb9f4112a7d275b8e413fb9c, 7a9824994cc942849265f899aa8c7bbc; no voters:
I20260929 03:10:54.203068 29221 raft_consensus.cc:2781] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:54.203138 29221 raft_consensus.cc:482] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:54.203179 29221 raft_consensus.cc:3037] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:54.203354 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3c710b333b45eb9b62bef317536b8e" candidate_uuid: "7a9824994cc942849265f899aa8c7bbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e19390aa37412e89ea4f0118263b24" is_pre_election: true
I20260929 03:10:54.203480 29142 raft_consensus.cc:2454] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9824994cc942849265f899aa8c7bbc in term 0.
I20260929 03:10:54.204398 29221 raft_consensus.cc:504] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.204725 29221 leader_election.cc:288] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [CANDIDATE]: Term 1 election: Requested vote from peers 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447), a5e19390aa37412e89ea4f0118263b24 (127.20.174.67:41375)
I20260929 03:10:54.205142 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3c710b333b45eb9b62bef317536b8e" candidate_uuid: "7a9824994cc942849265f899aa8c7bbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11536f7bfb9f4112a7d275b8e413fb9c"
I20260929 03:10:54.205287 29002 raft_consensus.cc:3037] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:54.205258 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3c710b333b45eb9b62bef317536b8e" candidate_uuid: "7a9824994cc942849265f899aa8c7bbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e19390aa37412e89ea4f0118263b24"
I20260929 03:10:54.205386 29142 raft_consensus.cc:3037] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:54.206488 29142 raft_consensus.cc:2454] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9824994cc942849265f899aa8c7bbc in term 1.
I20260929 03:10:54.206494 29002 raft_consensus.cc:2454] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9824994cc942849265f899aa8c7bbc in term 1.
I20260929 03:10:54.206805 28793 leader_election.cc:302] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9824994cc942849265f899aa8c7bbc, a5e19390aa37412e89ea4f0118263b24; no voters:
I20260929 03:10:54.207087 29221 raft_consensus.cc:2781] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:54.207311 29221 raft_consensus.cc:686] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 1 LEADER]: Becoming Leader. State: Replica: 7a9824994cc942849265f899aa8c7bbc, State: Running, Role: LEADER
I20260929 03:10:54.207576 29221 consensus_queue.cc:237] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.210471 28705 catalog_manager.cc:5495] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a9824994cc942849265f899aa8c7bbc (127.20.174.65). New cstate: current_term: 1 leader_uuid: "7a9824994cc942849265f899aa8c7bbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } health_report { overall_health: UNKNOWN } } }
I20260929 03:10:54.249892 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:54.245970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.20.174.65:44531)
0929 03:10:54.246247 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:54.246252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:54.246268 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:10:54.246359 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:54.246361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:54.246416 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:54.246606 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:54.246884 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.247396 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.248332 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.248568 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.248679 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:54.248984 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:54.249447 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:54.249581 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:54.249643 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:10:54.249705 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
I20260929 03:10:54.252825 29002 raft_consensus.cc:1261] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Refusing update from remote peer 7a9824994cc942849265f899aa8c7bbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:54.252796 29142 raft_consensus.cc:1261] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Refusing update from remote peer 7a9824994cc942849265f899aa8c7bbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:54.253402 29294 consensus_queue.cc:1035] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:54.253618 29221 consensus_queue.cc:1035] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:54.269544 28705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:54.269758 28705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:54.272514 28705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267: Sending DeleteTablet for 3 replicas of tablet 2d3c710b333b45eb9b62bef317536b8e
I20260929 03:10:54.273344 28982 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3c710b333b45eb9b62bef317536b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:54 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:51750
I20260929 03:10:54.273393 28842 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3c710b333b45eb9b62bef317536b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:54 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:42840
I20260929 03:10:54.273612 29122 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3c710b333b45eb9b62bef317536b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:54 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:36608
I20260929 03:10:54.273883 29275 tablet_replica.cc:330] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c: stopping tablet replica
I20260929 03:10:54.274040 29275 raft_consensus.cc:2229] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:54.274235 29275 raft_consensus.cc:2258] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:54.274498 29274 tablet_replica.cc:330] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc: stopping tablet replica
I20260929 03:10:54.274631 29274 raft_consensus.cc:2229] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:54.274991 29274 raft_consensus.cc:2258] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:54.275247 29276 tablet_replica.cc:330] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24: stopping tablet replica
I20260929 03:10:54.275401 29276 raft_consensus.cc:2229] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:54.275555 29276 raft_consensus.cc:2258] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:54.276449 29274 ts_tablet_manager.cc:1905] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:54.276676 29275 ts_tablet_manager.cc:1905] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:54.276971 28705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:10:54.277393 28705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:10:54.279147 29274 ts_tablet_manager.cc:1918] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:54.279241 29274 log.cc:1196] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-0/wal/wals/2d3c710b333b45eb9b62bef317536b8e
I20260929 03:10:54.279439 29275 ts_tablet_manager.cc:1918] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:54.279516 29274 ts_tablet_manager.cc:1939] T 2d3c710b333b45eb9b62bef317536b8e P 7a9824994cc942849265f899aa8c7bbc: Deleting consensus metadata
I20260929 03:10:54.279527 29275 log.cc:1196] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/2d3c710b333b45eb9b62bef317536b8e
I20260929 03:10:54.279693 29276 ts_tablet_manager.cc:1905] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:54.279822 29275 ts_tablet_manager.cc:1939] T 2d3c710b333b45eb9b62bef317536b8e P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting consensus metadata
I20260929 03:10:54.280689 28692 catalog_manager.cc:4841] TS 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531): tablet 2d3c710b333b45eb9b62bef317536b8e (table test-table [id=202c82bb912c4f8f88556cd739544bb8]) successfully deleted
I20260929 03:10:54.280920 28689 catalog_manager.cc:4841] TS 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447): tablet 2d3c710b333b45eb9b62bef317536b8e (table test-table [id=202c82bb912c4f8f88556cd739544bb8]) successfully deleted
I20260929 03:10:54.282039 29276 ts_tablet_manager.cc:1918] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:10:54.282143 29276 log.cc:1196] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/2d3c710b333b45eb9b62bef317536b8e
I20260929 03:10:54.282413 29276 ts_tablet_manager.cc:1939] T 2d3c710b333b45eb9b62bef317536b8e P a5e19390aa37412e89ea4f0118263b24: Deleting consensus metadata
I20260929 03:10:54.283241 28689 catalog_manager.cc:4841] TS a5e19390aa37412e89ea4f0118263b24 (127.20.174.67:41375): tablet 2d3c710b333b45eb9b62bef317536b8e (table test-table [id=202c82bb912c4f8f88556cd739544bb8]) successfully deleted
I20260929 03:10:54.284039 28842 tablet_service.cc:1467] Processing CreateTablet for tablet dfc4b6fad84342f0a57c0a790163fc5b (DEFAULT_TABLE table=test-table [id=b1063fae805a4a40a260e1a57c9f482b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:54.284337 28842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc4b6fad84342f0a57c0a790163fc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:54.284515 29122 tablet_service.cc:1467] Processing CreateTablet for tablet dfc4b6fad84342f0a57c0a790163fc5b (DEFAULT_TABLE table=test-table [id=b1063fae805a4a40a260e1a57c9f482b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:54.284665 28982 tablet_service.cc:1467] Processing CreateTablet for tablet dfc4b6fad84342f0a57c0a790163fc5b (DEFAULT_TABLE table=test-table [id=b1063fae805a4a40a260e1a57c9f482b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:10:54.284749 29122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc4b6fad84342f0a57c0a790163fc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:54.284927 28982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc4b6fad84342f0a57c0a790163fc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:54.287000 29216 tablet_bootstrap.cc:492] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc: Bootstrap starting.
I20260929 03:10:54.287349 29218 tablet_bootstrap.cc:492] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c: Bootstrap starting.
I20260929 03:10:54.288080 29216 tablet_bootstrap.cc:654] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:54.288108 29218 tablet_bootstrap.cc:654] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:54.288913 29217 tablet_bootstrap.cc:492] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24: Bootstrap starting.
I20260929 03:10:54.289841 29217 tablet_bootstrap.cc:654] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:54.290807 29216 tablet_bootstrap.cc:492] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc: No bootstrap required, opened a new log
I20260929 03:10:54.290908 29216 ts_tablet_manager.cc:1397] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:10:54.291405 29216 raft_consensus.cc:348] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.291538 29216 raft_consensus.cc:374] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:54.291574 29216 raft_consensus.cc:729] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9824994cc942849265f899aa8c7bbc, State: Initialized, Role: FOLLOWER
I20260929 03:10:54.291687 29218 tablet_bootstrap.cc:492] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c: No bootstrap required, opened a new log
I20260929 03:10:54.291785 29218 ts_tablet_manager.cc:1397] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 03:10:54.292092 29217 tablet_bootstrap.cc:492] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24: No bootstrap required, opened a new log
I20260929 03:10:54.292184 29217 ts_tablet_manager.cc:1397] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:54.292253 29218 raft_consensus.cc:348] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.292372 29218 raft_consensus.cc:374] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:54.292407 29218 raft_consensus.cc:729] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11536f7bfb9f4112a7d275b8e413fb9c, State: Initialized, Role: FOLLOWER
I20260929 03:10:54.292536 29218 consensus_queue.cc:260] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.292611 29217 raft_consensus.cc:348] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.292738 29217 raft_consensus.cc:374] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:54.292773 29217 raft_consensus.cc:729] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e19390aa37412e89ea4f0118263b24, State: Initialized, Role: FOLLOWER
I20260929 03:10:54.292814 29218 ts_tablet_manager.cc:1428] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:54.292902 29217 consensus_queue.cc:260] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.293169 29217 ts_tablet_manager.cc:1428] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:10:54.294389 29216 consensus_queue.cc:260] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.295274 29216 ts_tablet_manager.cc:1428] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 03:10:54.534332 29265 raft_consensus.cc:482] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:10:54.534572 29265 raft_consensus.cc:504] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.535255 29265 leader_election.cc:288] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531), 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447)
I20260929 03:10:54.535784 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc4b6fad84342f0a57c0a790163fc5b" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" is_pre_election: true
I20260929 03:10:54.535971 29002 raft_consensus.cc:2454] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 0.
I20260929 03:10:54.536057 28861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc4b6fad84342f0a57c0a790163fc5b" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9824994cc942849265f899aa8c7bbc" is_pre_election: true
I20260929 03:10:54.536294 28861 raft_consensus.cc:2454] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 0.
I20260929 03:10:54.536341 29073 leader_election.cc:302] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11536f7bfb9f4112a7d275b8e413fb9c, a5e19390aa37412e89ea4f0118263b24; no voters:
I20260929 03:10:54.536552 29265 raft_consensus.cc:2781] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:10:54.536614 29265 raft_consensus.cc:482] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:10:54.536665 29265 raft_consensus.cc:3037] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:54.537803 29265 raft_consensus.cc:504] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.538210 29265 leader_election.cc:288] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 election: Requested vote from peers 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531), 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447)
I20260929 03:10:54.538605 29002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc4b6fad84342f0a57c0a790163fc5b" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11536f7bfb9f4112a7d275b8e413fb9c"
I20260929 03:10:54.538759 29002 raft_consensus.cc:3037] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:54.538684 28861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc4b6fad84342f0a57c0a790163fc5b" candidate_uuid: "a5e19390aa37412e89ea4f0118263b24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9824994cc942849265f899aa8c7bbc"
I20260929 03:10:54.538887 28861 raft_consensus.cc:3037] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:54.539824 29002 raft_consensus.cc:2454] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 1.
I20260929 03:10:54.540023 28861 raft_consensus.cc:2454] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e19390aa37412e89ea4f0118263b24 in term 1.
I20260929 03:10:54.540140 29073 leader_election.cc:302] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11536f7bfb9f4112a7d275b8e413fb9c, a5e19390aa37412e89ea4f0118263b24; no voters:
I20260929 03:10:54.540331 29265 raft_consensus.cc:2781] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:54.540524 29265 raft_consensus.cc:686] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 1 LEADER]: Becoming Leader. State: Replica: a5e19390aa37412e89ea4f0118263b24, State: Running, Role: LEADER
I20260929 03:10:54.540760 29265 consensus_queue.cc:237] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } }
I20260929 03:10:54.542687 28700 catalog_manager.cc:5495] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e19390aa37412e89ea4f0118263b24 (127.20.174.67). New cstate: current_term: 1 leader_uuid: "a5e19390aa37412e89ea4f0118263b24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e19390aa37412e89ea4f0118263b24" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41375 } health_report { overall_health: HEALTHY } } }
I20260929 03:10:54.602645 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:54.599496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.20.174.67:41375)
0929 03:10:54.599633 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:54.599637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:54.599651 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:10:54.599773 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:54.599776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:54.599836 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:54.599912 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:10:54.600238 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.600743 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.601555 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:54.601776 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:54.601876 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:54.602132 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:10:54.602204 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:10:54.602258 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:54.602485 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 03:10:54.602538 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:10:54.609124 29002 raft_consensus.cc:1261] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Refusing update from remote peer a5e19390aa37412e89ea4f0118263b24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:54.609251 28861 raft_consensus.cc:1261] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Refusing update from remote peer a5e19390aa37412e89ea4f0118263b24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:10:54.609532 29305 consensus_queue.cc:1035] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11536f7bfb9f4112a7d275b8e413fb9c" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 35447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:54.609704 29265 consensus_queue.cc:1035] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9824994cc942849265f899aa8c7bbc" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:10:54.642867 28700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:54.643031 28700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:10:54.645212 28700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f81152bdbe6b4d54bfa9ca02161e0267: Sending DeleteTablet for 3 replicas of tablet dfc4b6fad84342f0a57c0a790163fc5b
I20260929 03:10:54.645944 28842 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc4b6fad84342f0a57c0a790163fc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:54 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:42840
I20260929 03:10:54.646057 29121 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc4b6fad84342f0a57c0a790163fc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:54 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:36608
I20260929 03:10:54.646237 29276 tablet_replica.cc:330] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24: stopping tablet replica
I20260929 03:10:54.646193 28982 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc4b6fad84342f0a57c0a790163fc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:10:54 UTC) from {username='oryx', principal='oryx/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:51750
I20260929 03:10:54.646368 29276 raft_consensus.cc:2229] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:10:54.646404 29274 tablet_replica.cc:330] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc: stopping tablet replica
I20260929 03:10:54.646528 29274 raft_consensus.cc:2229] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:54.646600 29276 raft_consensus.cc:2258] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:54.646651 29274 raft_consensus.cc:2258] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:54.646831 29275 tablet_replica.cc:330] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c: stopping tablet replica
I20260929 03:10:54.646965 29275 raft_consensus.cc:2229] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:10:54.647090 29275 raft_consensus.cc:2258] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:10:54.647498 29276 ts_tablet_manager.cc:1905] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:54.647575 29274 ts_tablet_manager.cc:1905] T dfc4b6fad84342f0a57c0a790163fc5b P 7a9824994cc942849265f899aa8c7bbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:54.648051 29275 ts_tablet_manager.cc:1905] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:10:54.648474 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 28777
I20260929 03:10:54.650345 29276 ts_tablet_manager.cc:1918] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:54.650506 29276 log.cc:1196] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/dfc4b6fad84342f0a57c0a790163fc5b
I20260929 03:10:54.650851 29276 ts_tablet_manager.cc:1939] T dfc4b6fad84342f0a57c0a790163fc5b P a5e19390aa37412e89ea4f0118263b24: Deleting consensus metadata
I20260929 03:10:54.651576 29275 ts_tablet_manager.cc:1918] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:10:54.651659 29275 log.cc:1196] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/dfc4b6fad84342f0a57c0a790163fc5b
I20260929 03:10:54.651744 28689 catalog_manager.cc:4841] TS a5e19390aa37412e89ea4f0118263b24 (127.20.174.67:41375): tablet dfc4b6fad84342f0a57c0a790163fc5b (table test-table [id=b1063fae805a4a40a260e1a57c9f482b]) successfully deleted
I20260929 03:10:54.651927 29275 ts_tablet_manager.cc:1939] T dfc4b6fad84342f0a57c0a790163fc5b P 11536f7bfb9f4112a7d275b8e413fb9c: Deleting consensus metadata
I20260929 03:10:54.652834 28689 catalog_manager.cc:4841] TS 11536f7bfb9f4112a7d275b8e413fb9c (127.20.174.66:35447): tablet dfc4b6fad84342f0a57c0a790163fc5b (table test-table [id=b1063fae805a4a40a260e1a57c9f482b]) successfully deleted
W20260929 03:10:54.656872 28692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:44531 (error 108)
I20260929 03:10:54.657145 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 28917
I20260929 03:10:54.658900 29313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:54.658379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:44531 (local address 127.0.0.1:42848)
0929 03:10:54.658680 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:54.658763 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:44531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":110,"threads_started":1}
W20260929 03:10:54.662603 28692 catalog_manager.cc:4584] TS 7a9824994cc942849265f899aa8c7bbc (127.20.174.65:44531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfc4b6fad84342f0a57c0a790163fc5b: Network error: Client connection negotiation failed: client connection to 127.20.174.65:44531: connect: Connection refused (error 111)
I20260929 03:10:54.665207 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 29057
I20260929 03:10:54.672487 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 28673
2026-09-29T03:10:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3653 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:54 dist-test-slave-k1dg krb5kdc[29316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:54 dist-test-slave-k1dg krb5kdc[29316](info): set up 2 sockets
Sep 29 03:10:54 dist-test-slave-k1dg krb5kdc[29316](info): commencing operation
krb5kdc: starting...
W20260929 03:10:55.779529 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:55 dist-test-slave-k1dg krb5kdc[29316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:55.914798 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40711
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:35939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:40711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:56.032320 29329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:56.032675 29329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:56.032758 29329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:56.037171 29329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:56.037295 29329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:56.037333 29329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:56.037369 29329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:56.037392 29329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:56.043430 29329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:35939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:40711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:56.044672 29329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:56.045830 29329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:56.055408 29337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:56.055454 29334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:56.055477 29329 server_base.cc:1034] running on GCE node
W20260929 03:10:56.055415 29335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:56.056231 29329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:56.057546 29329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:56.058765 29329 hybrid_clock.cc:648] HybridClock initialized: now 1790651456058738 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T03:10:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1398 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:10:56 dist-test-slave-k1dg krb5kdc[29342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:10:56 dist-test-slave-k1dg krb5kdc[29342](info): set up 2 sockets
Sep 29 03:10:56 dist-test-slave-k1dg krb5kdc[29342](info): commencing operation
krb5kdc: starting...
W20260929 03:10:57.123359 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:10:57 dist-test-slave-k1dg krb5kdc[29342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:10:57.258198 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44779
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:44779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:57.373643 29355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:57.373984 29355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:57.374066 29355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:57.378296 29355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:57.378414 29355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:57.378445 29355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:57.378499 29355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:57.378554 29355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:57.384198 29355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:44779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:57.385360 29355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:57.386497 29355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:57.395394 29360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:57.395408 29363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:57.395499 29361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:57.395891 29355 server_base.cc:1034] running on GCE node
I20260929 03:10:57.396560 29355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:57.397759 29355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:57.399020 29355 hybrid_clock.cc:648] HybridClock initialized: now 1790651457398997 us; error 40 us; skew 500 ppm
Sep 29 03:10:57 dist-test-slave-k1dg krb5kdc[29342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651457, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:57.402343 29355 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:57.403489 29355 webserver.cc:460] Webserver started at http://127.20.174.126:39663/ using document root <none> and password file <none>
I20260929 03:10:57.404119 29355 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:57.404215 29355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:57.404467 29355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:57.406795 29355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "77cbfc419bc2465e8f89fa55516a3100"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "8656e82f5321bde01d4131cb6b2a70b5"
server_key_iv: "c5171bf1877322e4b9b1b113672abb52"
server_key_version: "encryptionkey@0"
I20260929 03:10:57.407402 29355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "77cbfc419bc2465e8f89fa55516a3100"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "8656e82f5321bde01d4131cb6b2a70b5"
server_key_iv: "c5171bf1877322e4b9b1b113672abb52"
server_key_version: "encryptionkey@0"
I20260929 03:10:57.411667 29355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:10:57.414578 29370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:57.416023 29355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:57.416201 29355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "77cbfc419bc2465e8f89fa55516a3100"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "8656e82f5321bde01d4131cb6b2a70b5"
server_key_iv: "c5171bf1877322e4b9b1b113672abb52"
server_key_version: "encryptionkey@0"
I20260929 03:10:57.416350 29355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:57.433703 29355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:57.437252 29355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:57.437506 29355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:57.445325 29355 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:44779
I20260929 03:10:57.445371 29422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:44779 every 8 connection(s)
I20260929 03:10:57.446370 29355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:57.449438 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:10:57.455355 29423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100: Bootstrap starting.
I20260929 03:10:57.455536 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 29355
I20260929 03:10:57.455679 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:10:57.455957 21177 external_mini_cluster.cc:1428] Setting key ac7cc205790b97ca376b1be141005a9f
I20260929 03:10:57.458045 29423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100: Neither blocks nor log segments found. Creating new log.
I20260929 03:10:57.458878 29423 log.cc:824] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:57.460848 29423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100: No bootstrap required, opened a new log
I20260929 03:10:57.464480 29423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } }
I20260929 03:10:57.464712 29423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:10:57.464774 29423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cbfc419bc2465e8f89fa55516a3100, State: Initialized, Role: FOLLOWER
Sep 29 03:10:57 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:57.465256 29423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } }
I20260929 03:10:57.465384 29423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:57.465472 29423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:57.465575 29423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:10:57.466593 29423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } }
I20260929 03:10:57.466960 29423 leader_election.cc:302] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77cbfc419bc2465e8f89fa55516a3100; no voters:
I20260929 03:10:57.467305 29423 leader_election.cc:288] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:10:57.467384 29428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:10:57.467579 29428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 1 LEADER]: Becoming Leader. State: Replica: 77cbfc419bc2465e8f89fa55516a3100, State: Running, Role: LEADER
I20260929 03:10:57.467911 29428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } }
I20260929 03:10:57.468685 29423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:57.469718 29429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77cbfc419bc2465e8f89fa55516a3100. Latest consensus state: current_term: 1 leader_uuid: "77cbfc419bc2465e8f89fa55516a3100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } } }
I20260929 03:10:57.469726 29430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77cbfc419bc2465e8f89fa55516a3100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } } }
I20260929 03:10:57.469972 29430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:57.470288 29429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:57.470626 29437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:57.471887 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:57.457358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.20.174.126:44779)
0929 03:10:57.457765 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:57.457780 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:57.457823 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:10:57.458610 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:57.458629 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:57.459008 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:57.459374 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:57.460166 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:57.461231 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:57.462130 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:57.462692 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:57.462857 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:57.465567 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:57.465575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:57.465593 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:57.465626 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:57.467632 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:57.468211 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:57.468217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:57.468223 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:57.468361 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:57.468654 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:57.468659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:57.468661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:57.469191 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:57.470183 (+ 992us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:57.470366 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 03:10:57.470712 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":137,"threads_started":1}
I20260929 03:10:57.473731 29437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:57.479668 29437 catalog_manager.cc:1347] Generated new cluster ID: 4347df266d2b4050a78ac16218b70d70
I20260929 03:10:57.479743 29437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:57.499406 29437 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:10:57.500790 29437 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:57.513405 29437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100: Generated new TSK 0
I20260929 03:10:57.514271 29437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:10:57.551234 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44779
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:57.669149 29449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:57.669447 29449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:57.669529 29449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:57.674019 29449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:57.674103 29449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:57.674216 29449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:57.680155 29449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:57.681411 29449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:57.682912 29449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:57.693878 29455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:57.693982 29457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:57.693867 29454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:57.694028 29449 server_base.cc:1034] running on GCE node
I20260929 03:10:57.694718 29449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:57.695489 29449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:57.696682 29449 hybrid_clock.cc:648] HybridClock initialized: now 1790651457696644 us; error 50 us; skew 500 ppm
Sep 29 03:10:57 dist-test-slave-k1dg krb5kdc[29342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651457, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:57.699923 29449 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:10:57.700968 29449 webserver.cc:460] Webserver started at http://127.20.174.65:40779/ using document root <none> and password file <none>
I20260929 03:10:57.701598 29449 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:57.701702 29449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:57.701957 29449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:57.704213 29449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "a87e7667b9014a10a9c81ca807637464"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "7d6895bcf5c158d8e876182ff52e1d16"
server_key_iv: "3ee2e79ec3987ff3bbce373687f3fc22"
server_key_version: "encryptionkey@0"
I20260929 03:10:57.704818 29449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "a87e7667b9014a10a9c81ca807637464"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "7d6895bcf5c158d8e876182ff52e1d16"
server_key_iv: "3ee2e79ec3987ff3bbce373687f3fc22"
server_key_version: "encryptionkey@0"
I20260929 03:10:57.708926 29449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:10:57.711746 29464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:57.713270 29449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:57.713425 29449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "a87e7667b9014a10a9c81ca807637464"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "7d6895bcf5c158d8e876182ff52e1d16"
server_key_iv: "3ee2e79ec3987ff3bbce373687f3fc22"
server_key_version: "encryptionkey@0"
I20260929 03:10:57.713558 29449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:57.755790 29449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:57.759388 29449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:57.759651 29449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:57.760325 29449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:57.761241 29449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:57.761320 29449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:57.761397 29449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:57.761440 29449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:57.771220 29449 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:37249
I20260929 03:10:57.771294 29577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:37249 every 8 connection(s)
I20260929 03:10:57.772244 29449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:57.779340 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 29449
I20260929 03:10:57.779516 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:10:57.779834 21177 external_mini_cluster.cc:1428] Setting key 5742bf96dfeb72f2c25c3205df04373c
Sep 29 03:10:57 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651457, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:57.787423 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:57.774420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:33085 (local address 127.20.174.126:44779)
0929 03:10:57.774631 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:57.774635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:57.775716 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 03:10:57.776686 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:57.776689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:57.776738 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:57.776826 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:57.778209 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:57.778789 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:57.780220 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:57.780475 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:57.780567 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:57.783582 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:57.783586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:57.783588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:57.783614 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:57.785089 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:57.785684 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:57.785688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:57.785690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:57.785778 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:57.786166 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:57.786170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:57.786172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:57.786378 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:57.786594 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:57.787129 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 03:10:57.787257 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 03:10:57.788323 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:57.774791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:33085)
0929 03:10:57.775534 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:57.775582 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:57.776421 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:57.777008 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:57.777016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:57.777476 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:57.778039 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:57.778055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:57.778939 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:57.778942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:57.780076 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:57.780088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:57.780640 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:57.780648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:57.780840 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:57.781518 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:57.781536 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:57.783376 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:57.785261 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:57.785268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:57.785286 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:57.785570 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:57.785880 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:57.785883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:57.785885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:57.786054 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:57.786594 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:57.786602 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:57.786872 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 03:10:57.787079 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 03:10:57.787436 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:10:57.789712 29578 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44779
I20260929 03:10:57.790051 29578 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:57.790648 29578 heartbeater.cc:511] Master 127.20.174.126:44779 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:10:57.792411 29387 ts_manager.cc:194] Registered new tserver with Master: a87e7667b9014a10a9c81ca807637464 (127.20.174.65:37249)
I20260929 03:10:57.794095 29387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:33085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:10:57.811326 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44779
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:57.934280 29583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:57.934698 29583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:57.934801 29583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:57.939826 29583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:57.939916 29583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:57.940006 29583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:57.945923 29583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:57.947032 29583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:57.948087 29583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:57.958122 29588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:57.958244 29591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:57.958346 29583 server_base.cc:1034] running on GCE node
W20260929 03:10:57.958565 29589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:57.959091 29583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:57.959774 29583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:57.960968 29583 hybrid_clock.cc:648] HybridClock initialized: now 1790651457960950 us; error 28 us; skew 500 ppm
Sep 29 03:10:57 dist-test-slave-k1dg krb5kdc[29342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651457, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:57.963999 29583 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:10:57.964931 29583 webserver.cc:460] Webserver started at http://127.20.174.66:40621/ using document root <none> and password file <none>
I20260929 03:10:57.965516 29583 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:57.965612 29583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:57.965945 29583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:57.968226 29583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "9a42ea7f1b6042b5add4b3f549fdc915"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "8f37e1402241625da993636d2f1ba1fe"
server_key_iv: "f2d83274646b522f01a84b96afe08d7c"
server_key_version: "encryptionkey@0"
I20260929 03:10:57.968919 29583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "9a42ea7f1b6042b5add4b3f549fdc915"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "8f37e1402241625da993636d2f1ba1fe"
server_key_iv: "f2d83274646b522f01a84b96afe08d7c"
server_key_version: "encryptionkey@0"
I20260929 03:10:57.973435 29583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:10:57.976254 29598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:57.977669 29583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 03:10:57.977901 29583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "9a42ea7f1b6042b5add4b3f549fdc915"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "8f37e1402241625da993636d2f1ba1fe"
server_key_iv: "f2d83274646b522f01a84b96afe08d7c"
server_key_version: "encryptionkey@0"
I20260929 03:10:57.978041 29583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:57.995589 29583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:57.999321 29583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:57.999534 29583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:58.000212 29583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:58.001091 29583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:58.001147 29583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:58.001214 29583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:58.001245 29583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:58.010422 29583 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:43961
I20260929 03:10:58.010571 29711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:43961 every 8 connection(s)
I20260929 03:10:58.011495 29583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:58.018704 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 29583
I20260929 03:10:58.018836 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:10:58.019168 21177 external_mini_cluster.cc:1428] Setting key a51dcb6a086b487783b9494705318bd4
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651457, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:58.027343 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:58.013579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:52627 (local address 127.20.174.126:44779)
0929 03:10:58.013730 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:58.013734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:58.015007 (+ 1273us) server_negotiation.cc:407] Connection header received
0929 03:10:58.015975 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:58.015979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:58.016040 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:58.016149 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:58.017463 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.018009 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.019790 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.020116 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.020228 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:58.023328 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:58.023333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:58.023341 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:58.023369 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:58.025027 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:58.025617 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:58.025620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:58.025622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:58.025701 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:58.026085 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:58.026089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:58.026090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:58.026290 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:58.026402 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:58.027016 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 03:10:58.027159 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":55}
I20260929 03:10:58.028185 29714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:58.014005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:52627)
0929 03:10:58.014761 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:58.014815 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:58.015742 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:58.016334 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:58.016341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:58.016726 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:58.017305 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:58.017323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.018145 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.018155 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:58.019627 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:58.019638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.020308 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.020312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:58.020550 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:58.021334 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:58.021350 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:58.023130 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:58.025163 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:58.025170 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:58.025188 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:58.025490 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:58.025799 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:58.025802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:58.025804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:58.025975 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:58.026401 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:58.026409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:58.026710 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 03:10:58.026922 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 03:10:58.027287 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:10:58.029672 29712 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44779
I20260929 03:10:58.029961 29712 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:58.030602 29712 heartbeater.cc:511] Master 127.20.174.126:44779 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:10:58.031899 29387 ts_manager.cc:194] Registered new tserver with Master: 9a42ea7f1b6042b5add4b3f549fdc915 (127.20.174.66:43961)
I20260929 03:10:58.032822 29387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:52627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:10:58.051733 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44779
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:58.170024 29717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:58.170367 29717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:58.170419 29717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:58.174760 29717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:58.174837 29717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:58.174901 29717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:58.180475 29717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:44779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:58.181676 29717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:58.182905 29717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:58.192859 29722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:58.193094 29717 server_base.cc:1034] running on GCE node
W20260929 03:10:58.192938 29725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:58.192859 29723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:58.193686 29717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:58.194371 29717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:58.195621 29717 hybrid_clock.cc:648] HybridClock initialized: now 1790651458195599 us; error 33 us; skew 500 ppm
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651458, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:58.198746 29717 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:10:58.199810 29717 webserver.cc:460] Webserver started at http://127.20.174.67:35271/ using document root <none> and password file <none>
I20260929 03:10:58.200423 29717 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:58.200519 29717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:58.200764 29717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:10:58.203030 29717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "2d88d42dcd344edc927d9873bc4ac552"
format_stamp: "Formatted at 2026-09-29 03:10:58 on dist-test-slave-k1dg"
server_key: "f4fa1711f10f01f8ca137f53fbace784"
server_key_iv: "f7f67ecca16889ec9c83ed7de3980e31"
server_key_version: "encryptionkey@0"
I20260929 03:10:58.203624 29717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "2d88d42dcd344edc927d9873bc4ac552"
format_stamp: "Formatted at 2026-09-29 03:10:58 on dist-test-slave-k1dg"
server_key: "f4fa1711f10f01f8ca137f53fbace784"
server_key_iv: "f7f67ecca16889ec9c83ed7de3980e31"
server_key_version: "encryptionkey@0"
I20260929 03:10:58.207890 29717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 03:10:58.210559 29732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:58.211974 29717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:10:58.212121 29717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "2d88d42dcd344edc927d9873bc4ac552"
format_stamp: "Formatted at 2026-09-29 03:10:58 on dist-test-slave-k1dg"
server_key: "f4fa1711f10f01f8ca137f53fbace784"
server_key_iv: "f7f67ecca16889ec9c83ed7de3980e31"
server_key_version: "encryptionkey@0"
I20260929 03:10:58.212235 29717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:58.235458 29717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:58.238927 29717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:58.239157 29717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:58.239809 29717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:58.241199 29717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:58.241281 29717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:58.241349 29717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:58.241381 29717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:58.250698 29717 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:33553
I20260929 03:10:58.250736 29845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:33553 every 8 connection(s)
I20260929 03:10:58.251731 29717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:58.258891 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 29717
I20260929 03:10:58.259012 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:10:58.259384 21177 external_mini_cluster.cc:1428] Setting key ded03d3bdb252bd2e0395579d186cdae
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651458, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:10:58.266845 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:58.253897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:37105 (local address 127.20.174.126:44779)
0929 03:10:58.254046 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:58.254050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:58.255000 (+ 950us) server_negotiation.cc:407] Connection header received
0929 03:10:58.256376 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:58.256380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:58.256440 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:58.256524 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:58.257764 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.258319 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.259698 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.259980 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.260091 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:58.263275 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:58.263278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:58.263280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:58.263307 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:58.264724 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:58.265233 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:58.265235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:58.265237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:58.265314 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:58.265682 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:58.265684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:58.265685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:58.265862 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:58.265937 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:58.266612 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 03:10:58.266718 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:10:58.267840 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:58.254244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:37105)
0929 03:10:58.254807 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:58.254856 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:58.256122 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:58.256716 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:58.256723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:58.257103 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:58.257605 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:58.257620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.258484 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.258488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:58.259566 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:58.259577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.260192 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.260197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:58.260486 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:58.261287 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:58.261305 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:58.263100 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:10:58.264846 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:58.264851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:58.264865 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:58.265133 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:58.265426 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:10:58.265430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:10:58.265431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:10:58.265584 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:10:58.265983 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:10:58.265988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:10:58.266297 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 03:10:58.266558 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 03:10:58.266933 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":187,"threads_started":1}
I20260929 03:10:58.269207 29846 heartbeater.cc:347] Connected to a master server at 127.20.174.126:44779
I20260929 03:10:58.269562 29846 heartbeater.cc:464] Registering TS with master...
I20260929 03:10:58.270047 29846 heartbeater.cc:511] Master 127.20.174.126:44779 requested a full tablet report, sending...
I20260929 03:10:58.271324 29387 ts_manager.cc:194] Registered new tserver with Master: 2d88d42dcd344edc927d9873bc4ac552 (127.20.174.67:33553)
I20260929 03:10:58.271962 29387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:37105
I20260929 03:10:58.273774 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab.
Entry for principal oryx/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab.
WARNING: no policy specified for oryx/127.20.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab.
Entry for principal oryx/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab.
WARNING: no policy specified for oryx/127.20.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab.
Entry for principal oryx/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab.
I20260929 03:10:58.358286 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 29449
I20260929 03:10:58.363845 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 29583
I20260929 03:10:58.369705 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 29717
I20260929 03:10:58.375113 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 29355
I20260929 03:10:58.381695 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44779
--webserver_interface=127.20.174.126
--webserver_port=39663
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:44779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:10:58.497617 29855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:58.497979 29855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:58.498062 29855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:58.503640 29855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:10:58.503741 29855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:58.503780 29855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:58.503821 29855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:10:58.503856 29855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:10:58.509763 29855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:44779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=39663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:58.511037 29855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:58.512171 29855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:58.521073 29861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:58.521039 29863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:58.521345 29855 server_base.cc:1034] running on GCE node
W20260929 03:10:58.521126 29860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:58.522063 29855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:58.523365 29855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:58.524581 29855 hybrid_clock.cc:648] HybridClock initialized: now 1790651458524564 us; error 35 us; skew 500 ppm
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651458, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:58.527645 29855 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:10:58.528641 29855 webserver.cc:460] Webserver started at http://127.20.174.126:39663/ using document root <none> and password file <none>
I20260929 03:10:58.529224 29855 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:58.529301 29855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:58.533504 29855 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:58.536218 29870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:58.538000 29855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:58.538126 29855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "77cbfc419bc2465e8f89fa55516a3100"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "8656e82f5321bde01d4131cb6b2a70b5"
server_key_iv: "c5171bf1877322e4b9b1b113672abb52"
server_key_version: "encryptionkey@0"
I20260929 03:10:58.538829 29855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:58.569579 29855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:58.573060 29855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:58.573302 29855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:58.581317 29855 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:44779
I20260929 03:10:58.581360 29922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:44779 every 8 connection(s)
I20260929 03:10:58.582486 29855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:10:58.587275 29923 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:10:58.589097 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 29855
I20260929 03:10:58.589121 29923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100: Bootstrap starting.
I20260929 03:10:58.589617 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:37249
--local_ip_for_outbound_sockets=127.20.174.65
--tserver_master_addrs=127.20.174.126:44779
--webserver_port=40779
--webserver_interface=127.20.174.65
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.174.65
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:10:58.606353 29923 log.cc:824] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100: Log is configured to *not* fsync() on all Append() calls
I20260929 03:10:58.613399 29923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:10:58.613806 29923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100: Bootstrap complete.
I20260929 03:10:58.618208 29923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } }
I20260929 03:10:58.618745 29923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cbfc419bc2465e8f89fa55516a3100, State: Initialized, Role: FOLLOWER
I20260929 03:10:58.619299 29923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } }
I20260929 03:10:58.619451 29923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:10:58.619540 29923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:10:58.619652 29923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:10:58.621681 29923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } }
I20260929 03:10:58.622046 29923 leader_election.cc:302] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77cbfc419bc2465e8f89fa55516a3100; no voters:
I20260929 03:10:58.622414 29923 leader_election.cc:288] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:10:58.622556 29927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:10:58.622782 29927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [term 2 LEADER]: Becoming Leader. State: Replica: 77cbfc419bc2465e8f89fa55516a3100, State: Running, Role: LEADER
I20260929 03:10:58.623093 29927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } }
I20260929 03:10:58.623737 29923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:10:58.624502 29928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "77cbfc419bc2465e8f89fa55516a3100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } } }
I20260929 03:10:58.624632 29928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:58.625039 29929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77cbfc419bc2465e8f89fa55516a3100. Latest consensus state: current_term: 2 leader_uuid: "77cbfc419bc2465e8f89fa55516a3100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77cbfc419bc2465e8f89fa55516a3100" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44779 } } }
I20260929 03:10:58.625178 29929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100 [sys.catalog]: This master's current role is: LEADER
I20260929 03:10:58.625464 29936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:10:58.628964 29936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:10:58.629716 29936 catalog_manager.cc:1259] Loaded cluster ID: 4347df266d2b4050a78ac16218b70d70
I20260929 03:10:58.629791 29936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:10:58.632279 29936 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:10:58.633008 29936 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 77cbfc419bc2465e8f89fa55516a3100: Loaded TSK: 0
I20260929 03:10:58.634322 29936 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:10:58.713294 29925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:58.713573 29925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:58.713622 29925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:58.718072 29925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:58.718144 29925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:58.718243 29925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:10:58.724046 29925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.65.keytab
--principal=oryx/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:37249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=40779
--tserver_master_addrs=127.20.174.126:44779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:58.725332 29925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:58.726631 29925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:58.736783 29950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:58.736702 29949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:58.736694 29952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:58.737398 29925 server_base.cc:1034] running on GCE node
I20260929 03:10:58.737826 29925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:58.738590 29925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:58.739841 29925 hybrid_clock.cc:648] HybridClock initialized: now 1790651458739801 us; error 50 us; skew 500 ppm
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651458, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:58.743312 29925 init.cc:377] Logged in from keytab as oryx/127.20.174.65@KRBTEST.COM (short username oryx)
I20260929 03:10:58.744439 29925 webserver.cc:460] Webserver started at http://127.20.174.65:40779/ using document root <none> and password file <none>
I20260929 03:10:58.745069 29925 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:58.745184 29925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:58.749542 29925 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 03:10:58.752154 29959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:58.753640 29925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:58.753803 29925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "a87e7667b9014a10a9c81ca807637464"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "7d6895bcf5c158d8e876182ff52e1d16"
server_key_iv: "3ee2e79ec3987ff3bbce373687f3fc22"
server_key_version: "encryptionkey@0"
I20260929 03:10:58.754413 29925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:58.782640 29925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:58.785938 29925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:58.786162 29925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:58.786834 29925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:58.787747 29925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:58.787822 29925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:58.787895 29925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:58.787937 29925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:58.797626 29925 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:37249
I20260929 03:10:58.797647 30072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:37249 every 8 connection(s)
I20260929 03:10:58.798765 29925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:10:58.807204 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 29925
I20260929 03:10:58.807756 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:43961
--local_ip_for_outbound_sockets=127.20.174.66
--tserver_master_addrs=127.20.174.126:44779
--webserver_port=40621
--webserver_interface=127.20.174.66
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.174.66
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:10:58.814019 30075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:58.801236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:52959 (local address 127.20.174.126:44779)
0929 03:10:58.802346 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:58.802361 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:58.802400 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:10:58.803395 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:58.803407 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:58.803718 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:58.804031 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:58.805275 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.806352 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.807882 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.808740 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.808934 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:58.813263 (+ 4329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:52959: BlockingRecv error: recv got EOF from 127.20.174.65:52959 (error 108)
Metrics: {"server-negotiator.queue_time_us":964,"thread_start_us":152,"threads_started":1}
I20260929 03:10:58.814162 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:58.801249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:52959)
0929 03:10:58.802083 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:58.802139 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:58.803006 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:58.804128 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:58.804135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:58.804518 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:58.805071 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:58.805085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.806557 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.806561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:58.807723 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:58.807735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.808939 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.808943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:58.809146 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:58.809841 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:58.809870 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:58.813041 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":159,"threads_started":1}
W20260929 03:10:58.814806 30073 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:44779 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.174.126:44779: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:10:58.821661 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:58.815347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:51187)
0929 03:10:58.815556 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:58.815572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:58.815681 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:58.815956 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:58.815962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:58.816160 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:58.816308 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:58.816318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.817058 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.817063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:58.818105 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:58.818121 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.818753 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.818758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:58.818885 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:58.819363 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:58.819384 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:58.821529 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124}
W20260929 03:10:58.821792 30075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:58.815426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:51187 (local address 127.20.174.126:44779)
0929 03:10:58.815553 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:58.815557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:58.815639 (+ 82us) server_negotiation.cc:407] Connection header received
0929 03:10:58.815787 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:58.815789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:58.815831 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:58.815904 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:58.816429 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.816942 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.818257 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.818625 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.818713 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:58.821671 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:51187: BlockingRecv error: recv got EOF from 127.20.174.65:51187 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:10:58.827896 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:58.822303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:57143)
0929 03:10:58.822474 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:58.822488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:58.822556 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:58.822778 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:58.822781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:58.822941 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:58.823033 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:58.823039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.823701 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.823704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:58.824630 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:58.824640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.825198 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.825204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:58.825289 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:58.825796 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:58.825808 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:58.827750 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 03:10:58.827987 30075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:58.822387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:57143 (local address 127.20.174.126:44779)
0929 03:10:58.822532 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:58.822540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:58.822552 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:10:58.822634 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:58.822636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:58.822673 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:58.822739 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:58.823123 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.823602 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.824735 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:58.825026 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:58.825139 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:58.827872 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:57143: BlockingRecv error: recv got EOF from 127.20.174.65:57143 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 03:10:58.828238 30073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:10:58.926278 30077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:58.926597 30077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:58.926672 30077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:58.930964 30077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:58.931030 30077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:58.931124 30077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:10:58.936913 30077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.66.keytab
--principal=oryx/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:43961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=40621
--tserver_master_addrs=127.20.174.126:44779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:58.938133 30077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:58.939206 30077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:58.949635 30085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:58.949673 30077 server_base.cc:1034] running on GCE node
W20260929 03:10:58.949653 30083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:58.949637 30082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:58.950418 30077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:58.951159 30077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:58.952349 30077 hybrid_clock.cc:648] HybridClock initialized: now 1790651458952333 us; error 28 us; skew 500 ppm
Sep 29 03:10:58 dist-test-slave-k1dg krb5kdc[29342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651458, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:58.955493 30077 init.cc:377] Logged in from keytab as oryx/127.20.174.66@KRBTEST.COM (short username oryx)
I20260929 03:10:58.956401 30077 webserver.cc:460] Webserver started at http://127.20.174.66:40621/ using document root <none> and password file <none>
I20260929 03:10:58.957024 30077 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:58.957139 30077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:58.961354 30077 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:10:58.963577 30092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:58.964938 30077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:10:58.965092 30077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "9a42ea7f1b6042b5add4b3f549fdc915"
format_stamp: "Formatted at 2026-09-29 03:10:57 on dist-test-slave-k1dg"
server_key: "8f37e1402241625da993636d2f1ba1fe"
server_key_iv: "f2d83274646b522f01a84b96afe08d7c"
server_key_version: "encryptionkey@0"
I20260929 03:10:58.965648 30077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:58.984256 30077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:58.988178 30077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:58.988415 30077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:58.989104 30077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:58.990063 30077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:58.990139 30077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:58.990219 30077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:58.990264 30077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:59.000996 30077 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:43961
I20260929 03:10:59.001015 30205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:43961 every 8 connection(s)
I20260929 03:10:59.002200 30077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:10:59.005322 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 30077
I20260929 03:10:59.005872 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:33553
--local_ip_for_outbound_sockets=127.20.174.67
--tserver_master_addrs=127.20.174.126:44779
--webserver_port=35271
--webserver_interface=127.20.174.67
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.174.67
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:10:59.015863 30075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:59.004393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:48177 (local address 127.20.174.126:44779)
0929 03:10:59.004560 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:59.004564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:59.005828 (+ 1264us) server_negotiation.cc:407] Connection header received
0929 03:10:59.007388 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:59.007395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:59.007482 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:59.007606 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:59.009334 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.009881 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.011280 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.011504 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.011628 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.015652 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:48177: BlockingRecv error: recv got EOF from 127.20.174.66:48177 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:10:59.016322 30208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:59.004745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:48177)
0929 03:10:59.005619 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:59.005684 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:59.007067 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:59.007795 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:59.007803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:59.008308 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:59.009144 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.009174 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.010037 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.010041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.011114 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.011126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.011673 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.011677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.011868 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.012528 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:59.012547 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:59.015393 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":120,"threads_started":1}
W20260929 03:10:59.017035 30206 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:44779 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.174.126:44779: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:10:59.023065 30208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:59.017601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:37593)
0929 03:10:59.017726 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:59.017741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:59.017831 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:59.018080 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:59.018083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:59.018254 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:59.018343 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.018349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.019108 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.019114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.019669 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.019678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.020159 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.020162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.020260 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.020814 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:59.020826 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:59.022925 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 03:10:59.023236 30075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:59.017716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:37593 (local address 127.20.174.126:44779)
0929 03:10:59.017839 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:59.017843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:59.017859 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:10:59.017917 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:59.017919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:59.017970 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:59.018064 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:59.018486 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.018995 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.019814 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.020038 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.020153 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.023118 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:37593: BlockingRecv error: recv got EOF from 127.20.174.66:37593 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:10:59.029774 30208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:59.023946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:46315)
0929 03:10:59.024146 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:59.024160 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:59.024274 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:59.024561 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:59.024563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:59.024776 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:59.024853 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.024858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.025663 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.025666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.026210 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.026218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.026872 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.026877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.026998 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.027494 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:59.027510 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:59.029615 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133}
W20260929 03:10:59.029866 30075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:59.024019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:46315 (local address 127.20.174.126:44779)
0929 03:10:59.024148 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:59.024155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:59.024222 (+ 67us) server_negotiation.cc:407] Connection header received
0929 03:10:59.024396 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:59.024401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:59.024445 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:59.024560 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:59.024979 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.025539 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.026338 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.026735 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.026847 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.029744 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:46315: BlockingRecv error: recv got EOF from 127.20.174.66:46315 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 03:10:59.030262 30206 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:10:59.131203 30209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:10:59.131488 30209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:10:59.131533 30209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:10:59.136355 30209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:10:59.136440 30209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:10:59.136559 30209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:10:59.143311 30209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/krb5kdc/oryx_127.20.174.67.keytab
--principal=oryx/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:33553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=35271
--tserver_master_addrs=127.20.174.126:44779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:10:59.144639 30209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:10:59.145808 30209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:10:59.156781 30215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:59.156900 30217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:10:59.156781 30214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:10:59.157066 30209 server_base.cc:1034] running on GCE node
I20260929 03:10:59.157658 30209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:10:59.158553 30209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:10:59.159760 30209 hybrid_clock.cc:648] HybridClock initialized: now 1790651459159733 us; error 48 us; skew 500 ppm
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651459, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:10:59.162972 30209 init.cc:377] Logged in from keytab as oryx/127.20.174.67@KRBTEST.COM (short username oryx)
I20260929 03:10:59.164069 30209 webserver.cc:460] Webserver started at http://127.20.174.67:35271/ using document root <none> and password file <none>
I20260929 03:10:59.164749 30209 fs_manager.cc:359] Metadata directory not provided
I20260929 03:10:59.164873 30209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:10:59.169303 30209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:10:59.171804 30224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:59.173254 30209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:10:59.173424 30209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "2d88d42dcd344edc927d9873bc4ac552"
format_stamp: "Formatted at 2026-09-29 03:10:58 on dist-test-slave-k1dg"
server_key: "f4fa1711f10f01f8ca137f53fbace784"
server_key_iv: "f7f67ecca16889ec9c83ed7de3980e31"
server_key_version: "encryptionkey@0"
I20260929 03:10:59.174006 30209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:10:59.187449 30209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:10:59.190568 30209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:10:59.190793 30209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:10:59.191398 30209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:10:59.192253 30209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:10:59.192329 30209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:59.192396 30209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:10:59.192473 30209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:10:59.202212 30209 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:33553
I20260929 03:10:59.202232 30337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:33553 every 8 connection(s)
I20260929 03:10:59.203397 30209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:10:59.203745 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 30209
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:10:59.214419 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:59.204571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.20.174.126:44779)
0929 03:10:59.204742 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:59.204747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:59.205015 (+ 268us) server_negotiation.cc:407] Connection header received
0929 03:10:59.205117 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:59.205120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:59.205174 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:59.205275 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:59.206241 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.207059 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.208015 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.208277 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.208400 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.209932 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:10:59.209938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:10:59.209959 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:10:59.209988 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:10:59.211838 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:59.212371 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:59.212380 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:59.212390 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:59.212518 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:10:59.212842 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:10:59.212846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:10:59.212847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:10:59.213335 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:10:59.213567 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 03:10:59.213899 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 03:10:59.214212 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 03:10:59.215613 30341 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:59.206033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:39119 (local address 127.20.174.126:44779)
0929 03:10:59.206296 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:59.206300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:59.206727 (+ 427us) server_negotiation.cc:407] Connection header received
0929 03:10:59.208079 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:59.208083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:59.208135 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:59.208238 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:59.209494 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.210031 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.211311 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.211545 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.211660 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.215497 (+ 3837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:39119: BlockingRecv error: recv got EOF from 127.20.174.67:39119 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
I20260929 03:10:59.216305 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:59.206013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:39119)
0929 03:10:59.206586 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:59.206631 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:59.207849 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:59.208432 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:59.208456 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:59.208823 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:59.209333 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.209348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.210170 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.210174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.211155 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.211165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.211704 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.211708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.211883 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.212472 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:59.212489 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:59.215395 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":145,"threads_started":1}
W20260929 03:10:59.216825 30338 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:44779 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.174.126:44779: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:10:59.222443 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:59.217264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:49421)
0929 03:10:59.217365 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:59.217379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:59.217481 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:59.217715 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:59.217718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:59.217867 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:59.217994 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.218000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.218870 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.218873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.219415 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.219423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.219865 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.219868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.219945 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.220496 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:59.220507 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:59.222293 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 03:10:59.222559 30341 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:59.217343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:49421 (local address 127.20.174.126:44779)
0929 03:10:59.217468 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:59.217472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:59.217485 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:10:59.217562 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:59.217567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:59.217608 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:59.217682 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:59.218135 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.218764 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.219523 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.219732 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.219823 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.222421 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:49421: BlockingRecv error: recv got EOF from 127.20.174.67:49421 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:10:59.228439 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:59.223030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:58027)
0929 03:10:59.223143 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:59.223155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:59.223255 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:59.223533 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:59.223535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:59.223690 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:59.223761 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.223773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.224588 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.224593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.225273 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.225283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.225761 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.225773 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.225857 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.226379 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:59.226391 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:59.228296 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 03:10:59.228510 30341 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:59.223130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:58027 (local address 127.20.174.126:44779)
0929 03:10:59.223341 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:59.223345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:59.223357 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:10:59.223395 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:59.223397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:59.223431 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:59.223518 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:59.223883 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.224481 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.225397 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.225645 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.225758 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.228398 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:58027: BlockingRecv error: recv got EOF from 127.20.174.67:58027 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260929 03:10:59.228922 30338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:10:59 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:10:59.835356 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:10:59.828989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:45023)
0929 03:10:59.829295 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 03:10:59.829310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:10:59.829409 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:10:59.829711 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:10:59.829713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:10:59.829898 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:10:59.830047 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.830053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.830985 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.830988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.831561 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:10:59.831569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.832108 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.832110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:10:59.832186 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.832881 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:10:59.832895 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:10:59.835141 (+ 2246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":121,"threads_started":1}
W20260929 03:10:59.835398 30344 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:10:59.829156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:45023 (local address 127.20.174.126:44779)
0929 03:10:59.829397 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 03:10:59.829400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:10:59.829415 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:10:59.829543 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:10:59.829545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:10:59.829588 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:10:59.829668 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:10:59.830169 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.830853 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.831679 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:10:59.831996 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:10:59.832081 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:10:59.835273 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:45023: BlockingRecv error: recv got EOF from 127.20.174.65:45023 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
Sep 29 03:11:00 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:00 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:00.036618 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:00.031024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:43853)
0929 03:11:00.031294 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:00.031307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:00.031415 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:00.031648 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:00.031650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:00.031835 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:00.031929 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:00.031934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.032698 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.032700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:00.033297 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:00.033306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.033785 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.033788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:00.033890 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:00.034460 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:00.034472 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:00.036489 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":102,"threads_started":1}
W20260929 03:11:00.036779 30344 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:00.031076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:43853 (local address 127.20.174.126:44779)
0929 03:11:00.031255 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:00.031262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:00.031387 (+ 125us) server_negotiation.cc:407] Connection header received
0929 03:11:00.031496 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:00.031499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:00.031543 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:00.031615 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:00.032059 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.032589 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.033439 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.033669 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.033765 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:00.036641 (+ 2876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:43853: BlockingRecv error: recv got EOF from 127.20.174.66:43853 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 03:11:00 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:00 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:00.235837 30346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:00.229752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:50867)
0929 03:11:00.230090 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:00.230115 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:00.230219 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:00.230492 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:00.230494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:00.230677 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:00.230773 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:00.230779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.231546 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.231549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:00.232149 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:00.232158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.232635 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.232637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:00.232699 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:00.233335 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:00.233348 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:00.235697 (+ 2349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":75,"threads_started":1}
W20260929 03:11:00.235971 30344 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:00.230014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:50867 (local address 127.20.174.126:44779)
0929 03:11:00.230167 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:00.230171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:00.230187 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:11:00.230315 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:00.230317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:00.230365 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:00.230479 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:00.230891 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.231410 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.232271 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.232522 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.232608 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:00.235840 (+ 3232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:50867: BlockingRecv error: recv got EOF from 127.20.174.67:50867 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 03:11:00 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:00 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:00.842625 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:00.836221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:56971)
0929 03:11:00.836538 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:00.836552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:00.836645 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:00.836975 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:00.836978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:00.837167 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:00.837265 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:00.837270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.838177 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.838180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:00.838794 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:00.838803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.839289 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.839291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:00.839358 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:00.839968 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:00.839981 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:00.842440 (+ 2459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":134,"threads_started":1}
W20260929 03:11:00.842830 30348 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:00.836456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:56971 (local address 127.20.174.126:44779)
0929 03:11:00.836691 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:00.836695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:00.836707 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:11:00.836747 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:00.836750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:00.836802 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:00.836891 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:00.837414 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.838029 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.838926 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:00.839162 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:00.839266 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:00.842669 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:56971: BlockingRecv error: recv got EOF from 127.20.174.65:56971 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
Sep 29 03:11:01 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:01 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:01.044567 30349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:01.037623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:36837)
0929 03:11:01.038006 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:01.038024 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:01.038208 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:01.038550 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:01.038553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:01.038858 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:01.038959 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:01.038966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.039770 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.039775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:01.040470 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:01.040482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.041063 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.041067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:01.041193 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:01.041858 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:01.041873 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:01.044413 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":129,"threads_started":1}
W20260929 03:11:01.044785 30348 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:01.037819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:36837 (local address 127.20.174.126:44779)
0929 03:11:01.038007 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:01.038013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:01.038125 (+ 112us) server_negotiation.cc:407] Connection header received
0929 03:11:01.038339 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:01.038342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:01.038401 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:01.038527 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:01.039098 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.039636 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.040602 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.040871 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.040981 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:01.044599 (+ 3618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:36837: BlockingRecv error: recv got EOF from 127.20.174.66:36837 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 03:11:01 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:01 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:01.251646 30350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:01.236803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:45555)
0929 03:11:01.237109 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:01.237127 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:01.237244 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:01.237585 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:01.237589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:01.237896 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:01.238006 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:01.238014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.238965 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.238971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:01.239623 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:01.239632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.240159 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.240163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:01.240257 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:01.249378 (+ 9121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:01.249394 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:01.251494 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":119,"threads_started":1}
W20260929 03:11:01.251742 30348 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:01.237035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:45555 (local address 127.20.174.126:44779)
0929 03:11:01.237163 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:01.237171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:01.237187 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:11:01.237382 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:01.237387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:01.237452 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:01.237565 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:01.238152 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.238837 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.239766 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.240024 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.240152 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:01.251640 (+ 11488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:45555: BlockingRecv error: recv got EOF from 127.20.174.67:45555 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 03:11:01 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:01 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:01.850055 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:01.843591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:34747)
0929 03:11:01.843873 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:01.843887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:01.843988 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:01.844316 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:01.844319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:01.844503 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:01.844693 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:01.844708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.845741 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.845744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:01.846334 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:01.846343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.846877 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.846880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:01.846990 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:01.847516 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:01.847527 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:01.849853 (+ 2326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
W20260929 03:11:01.850142 30352 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:01.843732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:34747 (local address 127.20.174.126:44779)
0929 03:11:01.843996 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:01.844000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:01.844046 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:11:01.844086 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:01.844088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:01.844133 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:01.844224 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:01.845076 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.845618 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.846518 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:01.846751 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:01.846837 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:01.849991 (+ 3154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:34747: BlockingRecv error: recv got EOF from 127.20.174.65:34747 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
Sep 29 03:11:02 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:02 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:02.051306 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:02.045425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:46925)
0929 03:11:02.045684 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:02.045698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:02.045872 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:02.046101 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:02.046103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:02.046289 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:02.046389 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:02.046394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.047255 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.047258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:02.047896 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:02.047906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.048405 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.048408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:02.048475 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:02.049005 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:02.049018 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:02.051162 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
W20260929 03:11:02.051496 30352 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:02.045521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:46925 (local address 127.20.174.126:44779)
0929 03:11:02.045658 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:02.045662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:02.045838 (+ 176us) server_negotiation.cc:407] Connection header received
0929 03:11:02.045957 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:02.045959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:02.046002 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:02.046066 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:02.046558 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.047111 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.048018 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.048282 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.048365 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:02.051356 (+ 2991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:46925: BlockingRecv error: recv got EOF from 127.20.174.66:46925 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 03:11:02 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:02 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:02.258802 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:02.252715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:60137)
0929 03:11:02.253110 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:02.253122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:02.253261 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:02.253558 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:02.253561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:02.253861 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:02.253962 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:02.253968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.254836 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.254840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:02.255408 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:02.255416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.255933 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.255937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:02.256026 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:02.256605 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:02.256626 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:02.258668 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":105,"threads_started":1}
W20260929 03:11:02.259128 30352 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:02.252804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:60137 (local address 127.20.174.126:44779)
0929 03:11:02.252966 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:02.252973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:02.253209 (+ 236us) server_negotiation.cc:407] Connection header received
0929 03:11:02.253385 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:02.253392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:02.253442 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:02.253534 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:02.254099 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.254713 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.255546 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.255790 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.255926 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:02.258887 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:60137: BlockingRecv error: recv got EOF from 127.20.174.67:60137 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 03:11:02 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:02 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:02.858058 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:02.851194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:60237)
0929 03:11:02.851480 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:02.851511 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:02.851638 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:02.852233 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:02.852238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:02.852488 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:02.852648 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:02.852657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.853567 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.853570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:02.854654 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:02.854666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.855235 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.855240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:02.855327 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:02.855798 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:02.855809 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:02.857873 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
W20260929 03:11:02.858140 30356 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:02.851474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:60237 (local address 127.20.174.126:44779)
0929 03:11:02.851703 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:02.851707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:02.851719 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:11:02.851756 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:02.851758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:02.852042 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:02.852135 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:02.852784 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.853442 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.854788 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:02.855103 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:02.855215 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:02.858011 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:60237: BlockingRecv error: recv got EOF from 127.20.174.65:60237 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
Sep 29 03:11:03 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:03 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:03.058683 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:03.052146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:37541)
0929 03:11:03.052465 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:03.052480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:03.052602 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:03.052925 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:03.052929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:03.053157 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:03.053294 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:03.053302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.054214 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.054216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:03.054943 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:03.054955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.055478 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.055481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:03.055549 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:03.056184 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:03.056196 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:03.058411 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
W20260929 03:11:03.058856 30356 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:03.052232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:37541 (local address 127.20.174.126:44779)
0929 03:11:03.052375 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:03.052379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:03.052589 (+ 210us) server_negotiation.cc:407] Connection header received
0929 03:11:03.052707 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:03.052711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:03.052775 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:03.052876 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:03.053419 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.054085 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.055071 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.055359 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.055453 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:03.058670 (+ 3217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:37541: BlockingRecv error: recv got EOF from 127.20.174.66:37541 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 03:11:03 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:03 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:03.265661 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:03.259552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:52771)
0929 03:11:03.259939 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:03.259953 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:03.260052 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:03.260298 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:03.260300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:03.260490 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:03.260598 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:03.260604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.261438 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.261442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:03.262122 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:03.262133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.262753 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.262819 (+ 66us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:03.262901 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:03.263469 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:03.263482 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:03.265534 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":74,"threads_started":1}
W20260929 03:11:03.265931 30356 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:03.259633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:52771 (local address 127.20.174.126:44779)
0929 03:11:03.259793 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:03.259798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:03.260050 (+ 252us) server_negotiation.cc:407] Connection header received
0929 03:11:03.260123 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:03.260125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:03.260176 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:03.260276 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:03.260752 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.261296 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.262293 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.262622 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.262728 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:03.265723 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:52771: BlockingRecv error: recv got EOF from 127.20.174.67:52771 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 03:11:03 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:03 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:03.864881 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:03.858997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:60527)
0929 03:11:03.859327 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:03.859347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:03.859468 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:03.859736 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:03.859739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:03.860002 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:03.860101 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:03.860107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.860899 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.860902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:03.861498 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:03.861507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.861947 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.861950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:03.862017 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:03.862722 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:03.862735 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:03.864759 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":88,"threads_started":1}
W20260929 03:11:03.865120 30360 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:03.859184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:60527 (local address 127.20.174.126:44779)
0929 03:11:03.859494 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:03.859498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:03.859510 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:11:03.859548 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:03.859550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:03.859591 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:03.859671 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:03.860227 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.860758 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.861616 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:03.861836 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:03.861914 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:03.864966 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:60527: BlockingRecv error: recv got EOF from 127.20.174.65:60527 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
Sep 29 03:11:04 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:04 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:04.065799 30361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:04.059839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:49073)
0929 03:11:04.060119 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:04.060133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:04.060223 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:04.060522 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:04.060525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:04.060715 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:04.060830 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:04.060836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.061672 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.061676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:04.062319 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:04.062330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.062867 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.062870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:04.062959 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:04.063587 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:04.063600 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:04.065608 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
W20260929 03:11:04.065867 30360 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:04.060040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:49073 (local address 127.20.174.126:44779)
0929 03:11:04.060201 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:04.060205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:04.060219 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:11:04.060306 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:04.060308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:04.060353 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:04.060439 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:04.061000 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.061514 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.062488 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.062730 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.062860 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:04.065753 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:49073: BlockingRecv error: recv got EOF from 127.20.174.66:49073 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 03:11:04 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:04 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:04.273361 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:04.266925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:32831)
0929 03:11:04.267300 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:04.267324 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:04.267426 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:04.267787 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:04.267790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:04.268061 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:04.268191 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:04.268200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.269129 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.269135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:04.269725 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:04.269734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.270360 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.270363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:04.270487 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:04.271083 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:04.271095 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:04.273196 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":116,"threads_started":1}
W20260929 03:11:04.273589 30360 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:04.267127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:32831 (local address 127.20.174.126:44779)
0929 03:11:04.267294 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:04.267298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:04.267406 (+ 108us) server_negotiation.cc:407] Connection header received
0929 03:11:04.267566 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:04.267569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:04.267630 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:04.267763 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:04.268331 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.269001 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.269889 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.270224 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.270349 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:04.273377 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:32831: BlockingRecv error: recv got EOF from 127.20.174.67:32831 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 03:11:04 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:04 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:04.871697 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:04.865876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:37993)
0929 03:11:04.866207 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:04.866234 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:04.866338 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:04.866625 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:04.866627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:04.866826 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:04.866925 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:04.866931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.867732 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.867734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:04.868310 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:04.868319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.868925 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.868928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:04.869005 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:04.869527 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:04.869538 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:04.871575 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":91,"threads_started":1}
W20260929 03:11:04.871953 30364 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:04.866059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:37993 (local address 127.20.174.126:44779)
0929 03:11:04.866313 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:04.866317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:04.866331 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:11:04.866454 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:04.866457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:04.866500 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:04.866582 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:04.867048 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.867604 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.868445 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:04.868801 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:04.868892 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:04.871756 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:37993: BlockingRecv error: recv got EOF from 127.20.174.65:37993 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
Sep 29 03:11:05 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:05 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:05.073065 30365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:05.066823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:52771)
0929 03:11:05.067129 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:05.067157 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:05.067270 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:05.067594 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:05.067597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:05.067819 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:05.067930 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:05.067937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.068738 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.068741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:05.069432 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:05.069442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.069925 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.069928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:05.069995 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:05.070856 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:05.070876 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:05.072951 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":128,"threads_started":1}
W20260929 03:11:05.073302 30364 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:05.066895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:52771 (local address 127.20.174.126:44779)
0929 03:11:05.067105 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:05.067109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:05.067239 (+ 130us) server_negotiation.cc:407] Connection header received
0929 03:11:05.067369 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:05.067371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:05.067417 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:05.067511 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:05.068073 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.068613 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.069551 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.069775 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.069908 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:05.073184 (+ 3276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:52771: BlockingRecv error: recv got EOF from 127.20.174.66:52771 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 29 03:11:05 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:05 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:05.280653 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:05.274294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:55037)
0929 03:11:05.274607 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:05.274620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:05.274740 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:05.275073 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:05.275075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:05.275259 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:05.275357 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:05.275363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.276177 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.276180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:05.276743 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:05.276811 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.277412 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.277416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:05.277496 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:05.278133 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:05.278151 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:05.280537 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
W20260929 03:11:05.280853 30364 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:05.274498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:55037 (local address 127.20.174.126:44779)
0929 03:11:05.274653 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:05.274657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:05.274688 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:11:05.274906 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:05.274909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:05.274955 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:05.275027 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:05.275498 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.276063 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.276957 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.277287 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.277407 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:05.280682 (+ 3275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:55037: BlockingRecv error: recv got EOF from 127.20.174.67:55037 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 03:11:05 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:05 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:05.879076 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:05.872629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:40677)
0929 03:11:05.873042 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:05.873063 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:05.873234 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:05.873666 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:05.873670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:05.873900 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:05.874032 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:05.874039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.875087 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.875089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:05.875656 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:05.875665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.876145 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.876148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:05.876247 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:05.876780 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:05.876791 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:05.878955 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":77,"threads_started":1}
W20260929 03:11:05.879312 30368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:05.872765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:40677 (local address 127.20.174.126:44779)
0929 03:11:05.873064 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:05.873070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:05.873163 (+ 93us) server_negotiation.cc:407] Connection header received
0929 03:11:05.873345 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:05.873348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:05.873430 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:05.873555 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:05.874189 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.874950 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.875768 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:05.876007 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:05.876089 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:05.879162 (+ 3073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:40677: BlockingRecv error: recv got EOF from 127.20.174.65:40677 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":79,"threads_started":1}
Sep 29 03:11:06 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:06 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:06.080690 30369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:06.073858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:34301)
0929 03:11:06.074459 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:06.074478 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:06.074637 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:06.074967 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:06.074970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:06.075202 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:06.075344 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:06.075351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.076333 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.076340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:06.076979 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:06.076990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.077566 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.077570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:06.077665 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:06.078183 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:06.078195 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:06.080493 (+ 2298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":96,"threads_started":1}
W20260929 03:11:06.080793 30368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:06.074087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:34301 (local address 127.20.174.126:44779)
0929 03:11:06.074410 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:06.074416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:06.074571 (+ 155us) server_negotiation.cc:407] Connection header received
0929 03:11:06.074767 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:06.074771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:06.074833 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:06.074941 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:06.075488 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.076178 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.077131 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.077418 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.077547 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:06.080667 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:34301: BlockingRecv error: recv got EOF from 127.20.174.66:34301 (error 108)
Metrics: {"server-negotiator.queue_time_us":183}
Sep 29 03:11:06 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:06 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:06.295241 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:06.281693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:33907)
0929 03:11:06.282021 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:06.282034 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:06.282132 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:06.282588 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:06.282591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:06.282901 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:06.283021 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:06.283028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.283883 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.283886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:06.284513 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:06.284522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.284996 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.284999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:06.285072 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:06.292733 (+ 7661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:06.292756 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:06.295029 (+ 2273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
W20260929 03:11:06.295315 30368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:06.281863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:33907 (local address 127.20.174.126:44779)
0929 03:11:06.282017 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:06.282021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:06.282088 (+ 67us) server_negotiation.cc:407] Connection header received
0929 03:11:06.282222 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:06.282224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:06.282273 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:06.282383 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:06.283164 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.283712 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.284634 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.284867 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.284960 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:06.295212 (+ 10252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:33907: BlockingRecv error: recv got EOF from 127.20.174.67:33907 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 03:11:06 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:06 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:06.886971 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:06.879995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:34395)
0929 03:11:06.880322 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:06.880341 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:06.880455 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:06.881005 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:06.881008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:06.881257 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:06.881387 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:06.881395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.882357 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.882361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:06.883258 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:06.883269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.883889 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.883892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:06.883963 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:06.884525 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:06.884537 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:06.886826 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":143,"threads_started":1}
W20260929 03:11:06.887185 30372 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:06.880432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:34395 (local address 127.20.174.126:44779)
0929 03:11:06.880713 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:06.880717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:06.880730 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:11:06.880777 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:06.880780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:06.880824 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:06.880916 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:06.881577 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.882235 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.883420 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:06.883658 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:06.883741 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:06.887022 (+ 3281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:34395: BlockingRecv error: recv got EOF from 127.20.174.65:34395 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":128,"threads_started":1}
Sep 29 03:11:07 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:07 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:07.087230 30373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:07.081509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:59623)
0929 03:11:07.081782 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:07.081796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:07.081892 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:07.082112 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:07.082114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:07.082295 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:07.082390 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:07.082395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.083388 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.083391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:07.083973 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:07.083981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.084431 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.084434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:07.084527 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:07.085038 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:07.085050 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:07.087099 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
W20260929 03:11:07.087391 30372 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:07.081656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:59623 (local address 127.20.174.126:44779)
0929 03:11:07.081886 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:07.081890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:07.081904 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:11:07.081961 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:07.081963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:07.082003 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:07.082070 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:07.082554 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.083279 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.084086 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.084316 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.084399 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:07.087250 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:59623: BlockingRecv error: recv got EOF from 127.20.174.66:59623 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Sep 29 03:11:07 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:07 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:07.302803 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:07.296292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:44011)
0929 03:11:07.296691 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:07.296715 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:07.296864 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:07.297264 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:07.297267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:07.297489 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:07.297581 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:07.297587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.298366 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.298372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:07.299263 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:07.299274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.299814 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.299819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:07.299919 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:07.300546 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:07.300586 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:07.302674 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":97,"threads_started":1}
W20260929 03:11:07.302943 30372 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:07.296479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:44011 (local address 127.20.174.126:44779)
0929 03:11:07.296654 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:07.296660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:07.296807 (+ 147us) server_negotiation.cc:407] Connection header received
0929 03:11:07.297083 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:07.297087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:07.297136 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:07.297242 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:07.297714 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.298242 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.299397 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.299672 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.299783 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:07.302821 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:44011: BlockingRecv error: recv got EOF from 127.20.174.67:44011 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 03:11:07 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:07 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:07.894497 30375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:07.888027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:35479)
0929 03:11:07.888360 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:07.888411 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:07.888522 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:07.888933 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:07.888939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:07.889152 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:07.889254 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:07.889264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.890085 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.890088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:07.890744 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:07.890753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.891311 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.891314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:07.891389 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:07.892025 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:07.892037 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:07.894300 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":88,"threads_started":1}
W20260929 03:11:07.894832 30376 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:07.888073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:35479 (local address 127.20.174.126:44779)
0929 03:11:07.888521 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:07.888525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:07.888569 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:11:07.888633 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:07.888636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:07.888691 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:07.888804 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:07.889380 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.889972 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.890882 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:07.891141 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:07.891289 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:07.894678 (+ 3389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:35479: BlockingRecv error: recv got EOF from 127.20.174.65:35479 (error 108)
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":74,"threads_started":1}
Sep 29 03:11:08 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:08 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:08.094182 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:08.088146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:35573)
0929 03:11:08.088449 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:08.088466 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:08.088629 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:08.089036 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:08.089038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:08.089260 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:08.089358 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:08.089368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.090176 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.090181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:08.090837 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:08.090847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.091361 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.091365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:08.091453 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:08.091990 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:08.092002 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:08.093987 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
W20260929 03:11:08.094252 30376 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:08.088311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:35573 (local address 127.20.174.126:44779)
0929 03:11:08.088449 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:08.088455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:08.088560 (+ 105us) server_negotiation.cc:407] Connection header received
0929 03:11:08.088856 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:08.088860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:08.088912 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:08.089014 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:08.089485 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.090055 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.090974 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.091227 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.091344 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:08.094125 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:35573: BlockingRecv error: recv got EOF from 127.20.174.66:35573 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 03:11:08 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:08 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:08.310261 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:08.303894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:60517)
0929 03:11:08.304215 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:08.304230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:08.304387 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:08.304702 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:08.304704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:08.304894 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:08.305004 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:08.305011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.305829 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.305832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:08.306554 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:08.306567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.307105 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.307107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:08.307178 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:08.308024 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:08.308037 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:08.310118 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":137,"threads_started":1}
W20260929 03:11:08.310635 30376 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:08.304012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:60517 (local address 127.20.174.126:44779)
0929 03:11:08.304179 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:08.304184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:08.304357 (+ 173us) server_negotiation.cc:407] Connection header received
0929 03:11:08.304529 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:08.304531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:08.304579 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:08.304653 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:08.305142 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.305683 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.306695 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.306969 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.307074 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:08.310334 (+ 3260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:60517: BlockingRecv error: recv got EOF from 127.20.174.67:60517 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 03:11:08 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:08 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:08.901780 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:08.895642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:34009)
0929 03:11:08.896003 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:08.896020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:08.896160 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:08.896592 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:08.896595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:08.896790 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:08.896896 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:08.896902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.897703 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.897706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:08.898330 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:08.898340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.898902 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.898905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:08.898972 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:08.899520 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:08.899532 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:08.901642 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":88,"threads_started":1}
W20260929 03:11:08.902120 30380 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:08.895686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:34009 (local address 127.20.174.126:44779)
0929 03:11:08.895942 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:08.895948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:08.896115 (+ 167us) server_negotiation.cc:407] Connection header received
0929 03:11:08.896346 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:08.896352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:08.896417 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:08.896527 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:08.897039 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.897582 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.898541 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:08.898787 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:08.898874 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:08.901959 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:34009: BlockingRecv error: recv got EOF from 127.20.174.65:34009 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":62,"threads_started":1}
Sep 29 03:11:09 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:09 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:09.101518 30381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:09.095157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:56727)
0929 03:11:09.095457 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:09.095474 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:09.095585 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:09.095954 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:09.095956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:09.096162 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:09.096263 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:09.096268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.097112 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.097116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:09.097696 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:09.097706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.098219 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.098221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:09.098291 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:09.098944 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:09.098957 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:09.101393 (+ 2436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":123,"threads_started":1}
W20260929 03:11:09.102048 30380 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:09.095348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:56727 (local address 127.20.174.126:44779)
0929 03:11:09.095498 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:09.095502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:09.095535 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:11:09.095688 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:09.095691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:09.095749 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:09.095880 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:09.096412 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.096988 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.097823 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.098103 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.098188 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:09.101884 (+ 3696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:56727: BlockingRecv error: recv got EOF from 127.20.174.66:56727 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 03:11:09 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:09 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:09.317595 30382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:09.311343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:50523)
0929 03:11:09.311717 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:09.311732 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:09.311883 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:09.312267 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:09.312271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:09.312518 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:09.312646 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:09.312653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.313531 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.313533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:09.314172 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:09.314182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.314766 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.314769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:09.314858 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:09.315398 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:09.315412 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:09.317454 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":148,"threads_started":1}
W20260929 03:11:09.317837 30380 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:09.311512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:50523 (local address 127.20.174.126:44779)
0929 03:11:09.311786 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:09.311789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:09.311823 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:11:09.312053 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:09.312057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:09.312114 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:09.312193 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:09.312810 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.313412 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.314293 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.314632 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.314720 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:09.317689 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:50523: BlockingRecv error: recv got EOF from 127.20.174.67:50523 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 03:11:09 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:09 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:09.908974 30383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:09.902680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:60345)
0929 03:11:09.902960 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:09.902973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:09.903060 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:09.903326 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:09.903327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:09.903509 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:09.903603 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:09.903608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.904469 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.904472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:09.905118 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:09.905128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.905771 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.905774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:09.905929 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:09.906663 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:09.906704 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:09.908837 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":112,"threads_started":1}
W20260929 03:11:09.909264 30384 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:09.902824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:60345 (local address 127.20.174.126:44779)
0929 03:11:09.903082 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:09.903085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:09.903096 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:11:09.903131 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:09.903133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:09.903173 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:09.903261 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:09.903723 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.904318 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.905291 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:09.905602 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:09.905702 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:09.909114 (+ 3412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:60345: BlockingRecv error: recv got EOF from 127.20.174.65:60345 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":110,"threads_started":1}
Sep 29 03:11:10 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:10 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:10.108911 30385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:10.102391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:38839)
0929 03:11:10.102744 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:10.102759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:10.102912 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:10.103280 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:10.103282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:10.103487 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:10.103590 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:10.103596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.104499 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.104502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:10.105076 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:10.105084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.105608 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.105611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:10.105691 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:10.106290 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:10.106302 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:10.108706 (+ 2404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
W20260929 03:11:10.109169 30384 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:10.102550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:38839 (local address 127.20.174.126:44779)
0929 03:11:10.102688 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:10.102693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:10.103007 (+ 314us) server_negotiation.cc:407] Connection header received
0929 03:11:10.103045 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:10.103047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:10.103126 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:10.103213 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:10.103756 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.104373 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.105207 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.105459 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.105555 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:10.108927 (+ 3372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:38839: BlockingRecv error: recv got EOF from 127.20.174.66:38839 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 03:11:10 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:10 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:10.325130 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:10.318669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:50881)
0929 03:11:10.319112 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:10.319130 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:10.319297 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:10.319608 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:10.319610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:10.319888 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:10.319983 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:10.319988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.320777 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.320781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:10.321341 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:10.321348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.321827 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.321830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:10.321911 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:10.322526 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:10.322538 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:10.324917 (+ 2379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":158,"threads_started":1}
W20260929 03:11:10.325376 30384 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:10.318926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:50881 (local address 127.20.174.126:44779)
0929 03:11:10.319120 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:10.319126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:10.319231 (+ 105us) server_negotiation.cc:407] Connection header received
0929 03:11:10.319423 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:10.319427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:10.319484 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:10.319586 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:10.320119 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.320653 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.321472 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.321703 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.321823 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:10.325115 (+ 3292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:50881: BlockingRecv error: recv got EOF from 127.20.174.67:50881 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 29 03:11:10 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:10 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:10.917421 30387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:10.910008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:37331)
0929 03:11:10.910342 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:10.910401 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:10.910578 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:10.911198 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:10.911201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:10.911409 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:10.911518 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:10.911525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.912660 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.912664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:10.913511 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:10.913522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.914373 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.914376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:10.914541 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:10.915256 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:10.915303 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:10.917290 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":134,"threads_started":1}
W20260929 03:11:10.917699 30388 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:10.910228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:37331 (local address 127.20.174.126:44779)
0929 03:11:10.910548 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:10.910552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:10.910568 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:11:10.910956 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:10.910960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:10.911014 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:10.911111 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:10.911674 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.912475 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.913673 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:10.914184 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:10.914308 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:10.917506 (+ 3198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:37331: BlockingRecv error: recv got EOF from 127.20.174.65:37331 (error 108)
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
Sep 29 03:11:11 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:11 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:11.116060 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:11.109825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:46357)
0929 03:11:11.110192 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:11.110222 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:11.110382 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:11.110717 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:11.110720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:11.110922 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:11.111021 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:11.111026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.111874 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.111877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:11.112447 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:11.112456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.112991 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.112994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:11.113068 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:11.113603 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:11.113616 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:11.115926 (+ 2310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":77,"threads_started":1}
W20260929 03:11:11.116221 30388 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:11.110061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:46357 (local address 127.20.174.126:44779)
0929 03:11:11.110252 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:11.110259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:11.110302 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:11:11.110522 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:11.110525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:11.110572 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:11.110641 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:11.111144 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.111745 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.112569 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.112801 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.112888 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:11.116091 (+ 3203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:46357: BlockingRecv error: recv got EOF from 127.20.174.66:46357 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 03:11:11 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:11 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:11.339529 30390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:11.326344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:47311)
0929 03:11:11.326711 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:11.326725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:11.326866 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:11.327175 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:11.327178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:11.327376 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:11.327489 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:11.327495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.328411 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.328414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:11.329069 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:11.329080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.329596 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.329600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:11.329671 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:11.337001 (+ 7330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:11.337019 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:11.339391 (+ 2372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":179,"threads_started":1}
W20260929 03:11:11.339826 30388 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:11.326528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:47311 (local address 127.20.174.126:44779)
0929 03:11:11.326679 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:11.326683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:11.326846 (+ 163us) server_negotiation.cc:407] Connection header received
0929 03:11:11.326954 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:11.326957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:11.327010 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:11.327100 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:11.327646 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.328270 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.329222 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.329457 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.329546 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:11.339677 (+ 10131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:47311: BlockingRecv error: recv got EOF from 127.20.174.67:47311 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 03:11:11 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:11 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:11.924738 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:11.918398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:48851)
0929 03:11:11.918729 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:11.918749 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:11.918890 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:11.919243 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:11.919246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:11.919426 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:11.919526 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:11.919531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.920366 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.920370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:11.921006 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:11.921016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.921461 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.921464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:11.921527 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:11.922058 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:11.922072 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:11.924591 (+ 2519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":70,"threads_started":1}
W20260929 03:11:11.925034 30392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:11.918577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:48851 (local address 127.20.174.126:44779)
0929 03:11:11.918847 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:11.918851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:11.918864 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:11:11.919030 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:11.919032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:11.919076 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:11.919159 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:11.919650 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.920231 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.921125 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:11.921350 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:11.921428 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:11.924812 (+ 3384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:48851: BlockingRecv error: recv got EOF from 127.20.174.65:48851 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":186,"threads_started":1}
Sep 29 03:11:12 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:12 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:12.123835 30393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:12.117124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:52579)
0929 03:11:12.117609 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:12.117629 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:12.117829 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:12.118140 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:12.118143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:12.118419 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:12.118555 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:12.118561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.119438 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.119443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:12.120079 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:12.120088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.120623 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.120627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:12.120725 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:12.121286 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:12.121300 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:12.123642 (+ 2342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":119,"threads_started":1}
W20260929 03:11:12.123917 30392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:12.117358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:52579 (local address 127.20.174.126:44779)
0929 03:11:12.117616 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:12.117624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:12.117833 (+ 209us) server_negotiation.cc:407] Connection header received
0929 03:11:12.117940 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:12.117944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:12.118005 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:12.118127 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:12.118700 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.119293 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.120222 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.120486 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.120615 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:12.123795 (+ 3180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:52579: BlockingRecv error: recv got EOF from 127.20.174.66:52579 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
Sep 29 03:11:12 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:12 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:12.347014 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:12.340567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:55139)
0929 03:11:12.340957 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:12.340972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:12.341088 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:12.341408 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:12.341411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:12.341601 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:12.341719 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:12.341725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.342844 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.342847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:12.343499 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:12.343510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.344149 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.344153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:12.344222 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:12.344834 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:12.344847 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:12.346882 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":216,"threads_started":1}
W20260929 03:11:12.347234 30392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:12.340593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:55139 (local address 127.20.174.126:44779)
0929 03:11:12.340751 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:12.340755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:12.341090 (+ 335us) server_negotiation.cc:407] Connection header received
0929 03:11:12.341166 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:12.341169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:12.341225 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:12.341355 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:12.341926 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.342723 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.343635 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.343973 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.344086 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:12.347088 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:55139: BlockingRecv error: recv got EOF from 127.20.174.67:55139 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 03:11:12 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:12 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:12.932452 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:12.925817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:57401)
0929 03:11:12.926112 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:12.926133 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:12.926274 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:12.926630 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:12.926632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:12.926874 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:12.926983 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:12.926989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.927954 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.927958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:12.928873 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:12.928889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.929591 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.929594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:12.929659 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:12.930252 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:12.930265 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:12.932327 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
W20260929 03:11:12.932734 30396 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:12.925964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:57401 (local address 127.20.174.126:44779)
0929 03:11:12.926328 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:12.926332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:12.926347 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:11:12.926391 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:12.926393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:12.926477 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:12.926561 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:12.927119 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.927857 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.929082 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:12.929475 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:12.929567 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:12.932583 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:57401: BlockingRecv error: recv got EOF from 127.20.174.65:57401 (error 108)
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":84,"threads_started":1}
Sep 29 03:11:13 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:13 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:13.130903 30397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:13.124890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:34271)
0929 03:11:13.125238 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:13.125276 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:13.125405 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:13.125700 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:13.125703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:13.125984 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:13.126083 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:13.126089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.126900 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.126903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:13.127505 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:13.127514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.127979 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.127983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:13.128054 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:13.128584 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:13.128596 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:13.130710 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":148,"threads_started":1}
W20260929 03:11:13.130962 30396 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:13.124941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:34271 (local address 127.20.174.126:44779)
0929 03:11:13.125096 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:13.125101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:13.125372 (+ 271us) server_negotiation.cc:407] Connection header received
0929 03:11:13.125500 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:13.125503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:13.125548 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:13.125619 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:13.126215 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.126775 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.127622 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.127853 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.127933 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:13.130846 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:34271: BlockingRecv error: recv got EOF from 127.20.174.66:34271 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 03:11:13 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:13 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.67@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:13.353902 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:13.347872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.67:40103)
0929 03:11:13.348178 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:13.348216 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:13.348326 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:13.348649 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:13.348652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:13.348866 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:13.348964 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:13.348970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.349760 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.349764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:13.350405 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:13.350415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.351014 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.351017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:13.351092 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:13.351658 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:13.351671 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:13.353727 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":126,"threads_started":1}
W20260929 03:11:13.354020 30396 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:13.348091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:40103 (local address 127.20.174.126:44779)
0929 03:11:13.348222 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:13.348225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:13.348289 (+ 64us) server_negotiation.cc:407] Connection header received
0929 03:11:13.348407 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:13.348410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:13.348490 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:13.348567 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:13.349086 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.349633 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.350621 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.350868 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.350956 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:13.353887 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.67:40103: BlockingRecv error: recv got EOF from 127.20.174.67:40103 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 03:11:13 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:13 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.65@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:13.939973 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:13.933403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.65:45209)
0929 03:11:13.933704 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:13.933721 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:13.933870 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:13.934251 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:13.934254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:13.934524 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:13.934634 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:13.934640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.935425 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.935430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:13.936013 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:13.936021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.936523 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.936527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:13.936615 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:13.937250 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:13.937261 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:13.939781 (+ 2520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
W20260929 03:11:13.940078 30400 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:13.933479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:45209 (local address 127.20.174.126:44779)
0929 03:11:13.933723 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:13.933729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:13.933806 (+ 77us) server_negotiation.cc:407] Connection header received
0929 03:11:13.934064 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:13.934068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:13.934124 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:13.934226 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:13.934754 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.935304 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.936142 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:13.936389 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:13.936517 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:13.939928 (+ 3411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.65:45209: BlockingRecv error: recv got EOF from 127.20.174.65:45209 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
Sep 29 03:11:14 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:11:14 dist-test-slave-k1dg krb5kdc[29342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.174.66@KRBTEST.COM for oryx/127.20.174.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:11:14.137980 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:14.132055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:44779 (local address 127.20.174.66:56443)
0929 03:11:14.132380 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:14.132393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:14.132538 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:14.132848 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:14.132852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:14.133069 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:14.133178 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:14.133184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.134051 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.134056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:14.134755 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:14.134767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.135267 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.135271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:14.135368 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:14.135840 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:14.135851 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:14.137852 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.174.126:44779: Server oryx/127.20.174.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":88,"threads_started":1}
W20260929 03:11:14.138134 30400 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:14.132122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:56443 (local address 127.20.174.126:44779)
0929 03:11:14.132327 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:14.132332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:14.132475 (+ 143us) server_negotiation.cc:407] Connection header received
0929 03:11:14.132658 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:14.132661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:14.132727 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:14.132829 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:14.133374 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.133932 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.134889 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.135125 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.135240 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:14.138011 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.174.66:56443: BlockingRecv error: recv got EOF from 127.20.174.66:56443 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 03:11:14.204543 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 29925
I20260929 03:11:14.210683 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 30077
I20260929 03:11:14.216598 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 30209
I20260929 03:11:14.222107 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 29855
2026-09-29T03:11:14Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18148 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T03:11:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:11:14Z Disabled control of system clock
I20260929 03:11:14.253373 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:34353
--webserver_interface=127.20.174.126
--webserver_port=0
--builtin_ntp_servers=127.20.174.84:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:34353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:11:14.375404 30405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:14.375744 30405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:14.375823 30405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:14.380647 30405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:11:14.380720 30405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:14.380738 30405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:14.380757 30405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:11:14.380774 30405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:11:14.386206 30405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:34353
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:34353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:14.387445 30405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:14.388942 30405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:14.398649 30410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:14.398690 30413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:14.398828 30405 server_base.cc:1034] running on GCE node
W20260929 03:11:14.398663 30411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:14.399616 30405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:14.401199 30405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:14.402400 30405 hybrid_clock.cc:648] HybridClock initialized: now 1790651474402384 us; error 37 us; skew 500 ppm
I20260929 03:11:14.404629 30405 webserver.cc:460] Webserver started at http://127.20.174.126:33729/ using document root <none> and password file <none>
I20260929 03:11:14.405284 30405 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:14.405349 30405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:14.405586 30405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:14.407888 30405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "3e006161b626471394859d2be46930ee"
format_stamp: "Formatted at 2026-09-29 03:11:14 on dist-test-slave-k1dg"
server_key: "6901a51c16752a1c2404947d65faaeb9"
server_key_iv: "635d9bd52542da3431ab041f3f29bfdf"
server_key_version: "encryptionkey@0"
I20260929 03:11:14.408500 30405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "3e006161b626471394859d2be46930ee"
format_stamp: "Formatted at 2026-09-29 03:11:14 on dist-test-slave-k1dg"
server_key: "6901a51c16752a1c2404947d65faaeb9"
server_key_iv: "635d9bd52542da3431ab041f3f29bfdf"
server_key_version: "encryptionkey@0"
I20260929 03:11:14.412812 30405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:11:14.415881 30419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:14.417481 30405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:11:14.417662 30405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "3e006161b626471394859d2be46930ee"
format_stamp: "Formatted at 2026-09-29 03:11:14 on dist-test-slave-k1dg"
server_key: "6901a51c16752a1c2404947d65faaeb9"
server_key_iv: "635d9bd52542da3431ab041f3f29bfdf"
server_key_version: "encryptionkey@0"
I20260929 03:11:14.417812 30405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:14.436266 30405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:14.437044 30405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:14.437259 30405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:14.445096 30405 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:34353
I20260929 03:11:14.445117 30471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:34353 every 8 connection(s)
I20260929 03:11:14.446187 30405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:11:14.449954 30472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:11:14.450544 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 30405
I20260929 03:11:14.450683 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:11:14.451007 21177 external_mini_cluster.cc:1428] Setting key 432b8f363c5f00360e2ebe574fd08493
I20260929 03:11:14.456504 30472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee: Bootstrap starting.
I20260929 03:11:14.459105 30472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee: Neither blocks nor log segments found. Creating new log.
I20260929 03:11:14.459933 30472 log.cc:824] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee: Log is configured to *not* fsync() on all Append() calls
I20260929 03:11:14.461374 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:14.452519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.20.174.126:34353)
0929 03:11:14.453387 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:14.453406 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:14.453447 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:11:14.454117 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:14.454135 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:14.454673 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:14.455132 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:14.455579 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.456560 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.457566 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.458096 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.458286 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:14.459261 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:14.459271 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:14.459289 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:11:14.459746 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:14.459805 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:11:14.459838 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:14.459959 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:14.460125 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 03:11:14.460441 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":146,"threads_started":1}
I20260929 03:11:14.462464 30472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee: No bootstrap required, opened a new log
I20260929 03:11:14.466286 30472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e006161b626471394859d2be46930ee" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34353 } }
I20260929 03:11:14.466619 30472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:11:14.466663 30472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e006161b626471394859d2be46930ee, State: Initialized, Role: FOLLOWER
I20260929 03:11:14.467068 30472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e006161b626471394859d2be46930ee" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34353 } }
I20260929 03:11:14.467182 30472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:11:14.467223 30472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:11:14.467295 30472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:11:14.468297 30472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e006161b626471394859d2be46930ee" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34353 } }
I20260929 03:11:14.468729 30472 leader_election.cc:302] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e006161b626471394859d2be46930ee; no voters:
I20260929 03:11:14.469161 30472 leader_election.cc:288] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:11:14.469247 30477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:11:14.469493 30477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [term 1 LEADER]: Becoming Leader. State: Replica: 3e006161b626471394859d2be46930ee, State: Running, Role: LEADER
I20260929 03:11:14.469842 30477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e006161b626471394859d2be46930ee" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34353 } }
I20260929 03:11:14.470666 30472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:11:14.471736 30478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e006161b626471394859d2be46930ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e006161b626471394859d2be46930ee" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34353 } } }
I20260929 03:11:14.471769 30479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e006161b626471394859d2be46930ee. Latest consensus state: current_term: 1 leader_uuid: "3e006161b626471394859d2be46930ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e006161b626471394859d2be46930ee" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34353 } } }
I20260929 03:11:14.471880 30478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [sys.catalog]: This master's current role is: LEADER
I20260929 03:11:14.471891 30479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee [sys.catalog]: This master's current role is: LEADER
I20260929 03:11:14.472501 30486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:11:14.476027 30486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:11:14.481730 30486 catalog_manager.cc:1347] Generated new cluster ID: 3c69356bb3d843628713325ef08070a5
I20260929 03:11:14.481822 30486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:11:14.493479 30486 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:11:14.494796 30486 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:11:14.512863 30486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e006161b626471394859d2be46930ee: Generated new TSK 0
I20260929 03:11:14.513796 30486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:11:14.524072 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34353
--builtin_ntp_servers=127.20.174.84:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 03:11:14.641265 30496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:14.641561 30496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:14.641601 30496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:14.646124 30496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:14.646199 30496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:14.646258 30496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:11:14.651386 30496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:14.652450 30496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:14.653816 30496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:14.663576 30501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:14.663599 30504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:14.663576 30502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:14.663976 30496 server_base.cc:1034] running on GCE node
I20260929 03:11:14.664427 30496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:14.665098 30496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:14.666285 30496 hybrid_clock.cc:648] HybridClock initialized: now 1790651474666263 us; error 31 us; skew 500 ppm
I20260929 03:11:14.668411 30496 webserver.cc:460] Webserver started at http://127.20.174.65:43219/ using document root <none> and password file <none>
I20260929 03:11:14.669025 30496 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:14.669092 30496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:14.669312 30496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:14.671692 30496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "0aab3befc8f849eda000f0cfac3c5d9d"
format_stamp: "Formatted at 2026-09-29 03:11:14 on dist-test-slave-k1dg"
server_key: "e8d8c82f97c314aa2688e8f26b11381a"
server_key_iv: "9475a289737fba2871e1839105253304"
server_key_version: "encryptionkey@0"
I20260929 03:11:14.672364 30496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "0aab3befc8f849eda000f0cfac3c5d9d"
format_stamp: "Formatted at 2026-09-29 03:11:14 on dist-test-slave-k1dg"
server_key: "e8d8c82f97c314aa2688e8f26b11381a"
server_key_iv: "9475a289737fba2871e1839105253304"
server_key_version: "encryptionkey@0"
I20260929 03:11:14.676446 30496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:11:14.679553 30510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:14.681061 30496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:11:14.681223 30496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "0aab3befc8f849eda000f0cfac3c5d9d"
format_stamp: "Formatted at 2026-09-29 03:11:14 on dist-test-slave-k1dg"
server_key: "e8d8c82f97c314aa2688e8f26b11381a"
server_key_iv: "9475a289737fba2871e1839105253304"
server_key_version: "encryptionkey@0"
I20260929 03:11:14.681358 30496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:14.705693 30496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:14.706527 30496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:14.706733 30496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:14.707361 30496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:11:14.708376 30496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:11:14.708451 30496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:14.708531 30496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:11:14.708585 30496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:14.717540 30496 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:40503
I20260929 03:11:14.717578 30623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:40503 every 8 connection(s)
I20260929 03:11:14.718600 30496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:11:14.720911 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 30496
I20260929 03:11:14.721028 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:11:14.721324 21177 external_mini_cluster.cc:1428] Setting key c2f2e205bde93e800ca2c2d8413b1230
I20260929 03:11:14.723510 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34353
--builtin_ntp_servers=127.20.174.84:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 03:11:14.729991 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:14.720533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:44201 (local address 127.20.174.126:34353)
0929 03:11:14.720710 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:14.720717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:14.722175 (+ 1458us) server_negotiation.cc:407] Connection header received
0929 03:11:14.723849 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:14.723856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:14.723927 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:14.724031 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:14.725156 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.725699 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.727678 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.727960 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.728055 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:14.729461 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:14.729464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:14.729466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:11:14.729478 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:14.729497 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:11:14.729505 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:14.729564 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:14.729772 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 03:11:14.729876 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:11:14.731343 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:14.721145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:34353 (local address 127.20.174.65:44201)
0929 03:11:14.722044 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:14.722088 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:14.723538 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:14.724194 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:14.724202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:14.724294 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:14.724976 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:14.724999 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.725864 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.725868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:14.727492 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:14.727506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.728137 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.728143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:14.728403 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:14.729107 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:11:14.729112 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:14.729175 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:11:14.729277 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:11:14.729599 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:11:14.729611 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:11:14.729620 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 03:11:14.729772 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 03:11:14.730123 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":161,"threads_started":1}
I20260929 03:11:14.732788 30624 heartbeater.cc:347] Connected to a master server at 127.20.174.126:34353
I20260929 03:11:14.733095 30624 heartbeater.cc:464] Registering TS with master...
I20260929 03:11:14.733639 30624 heartbeater.cc:511] Master 127.20.174.126:34353 requested a full tablet report, sending...
I20260929 03:11:14.735397 30436 ts_manager.cc:194] Registered new tserver with Master: 0aab3befc8f849eda000f0cfac3c5d9d (127.20.174.65:40503)
I20260929 03:11:14.736661 30436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.65:44201
W20260929 03:11:14.840003 30627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:14.840346 30627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:14.840428 30627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:14.844614 30627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:14.844720 30627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:14.844818 30627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:11:14.849696 30627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:14.850898 30627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:14.852375 30627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:14.862341 30635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:14.862360 30633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:14.862331 30632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:14.862671 30627 server_base.cc:1034] running on GCE node
I20260929 03:11:14.863245 30627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:14.863879 30627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:14.865118 30627 hybrid_clock.cc:648] HybridClock initialized: now 1790651474865098 us; error 34 us; skew 500 ppm
I20260929 03:11:14.867151 30627 webserver.cc:460] Webserver started at http://127.20.174.66:45197/ using document root <none> and password file <none>
I20260929 03:11:14.867784 30627 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:14.867882 30627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:14.868117 30627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:14.870342 30627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "5281e6fa2e44430b89bf1d332aa8302d"
format_stamp: "Formatted at 2026-09-29 03:11:14 on dist-test-slave-k1dg"
server_key: "7de6ffab65a98b68b588a57e67e647bb"
server_key_iv: "ff3749da78cfb078683403c0a12bfc3b"
server_key_version: "encryptionkey@0"
I20260929 03:11:14.871026 30627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "5281e6fa2e44430b89bf1d332aa8302d"
format_stamp: "Formatted at 2026-09-29 03:11:14 on dist-test-slave-k1dg"
server_key: "7de6ffab65a98b68b588a57e67e647bb"
server_key_iv: "ff3749da78cfb078683403c0a12bfc3b"
server_key_version: "encryptionkey@0"
I20260929 03:11:14.875123 30627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:11:14.877846 30641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:14.879189 30627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:11:14.879364 30627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "5281e6fa2e44430b89bf1d332aa8302d"
format_stamp: "Formatted at 2026-09-29 03:11:14 on dist-test-slave-k1dg"
server_key: "7de6ffab65a98b68b588a57e67e647bb"
server_key_iv: "ff3749da78cfb078683403c0a12bfc3b"
server_key_version: "encryptionkey@0"
I20260929 03:11:14.879487 30627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:14.897743 30627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:14.898550 30627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:14.898754 30627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:14.899367 30627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:11:14.900345 30627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:11:14.900408 30627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:14.900465 30627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:11:14.900488 30627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:14.909205 30627 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:44383
I20260929 03:11:14.909227 30754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:44383 every 8 connection(s)
I20260929 03:11:14.910148 30627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:11:14.910719 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 30627
I20260929 03:11:14.910840 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:11:14.911216 21177 external_mini_cluster.cc:1428] Setting key 57ccd5814f83a1429fa28f544dcc6d91
I20260929 03:11:14.913839 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34353
--builtin_ntp_servers=127.20.174.84:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 03:11:14.920957 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:14.912510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:51871 (local address 127.20.174.126:34353)
0929 03:11:14.912655 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:14.912660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:14.914107 (+ 1447us) server_negotiation.cc:407] Connection header received
0929 03:11:14.915899 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:14.915904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:14.915973 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:14.916085 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:14.917049 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.917599 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.918901 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.919133 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.919254 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:14.920411 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:14.920415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:14.920419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:11:14.920446 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:14.920466 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:11:14.920475 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:14.920554 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:14.920722 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:11:14.920806 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:11:14.921869 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:14.913110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:34353 (local address 127.20.174.66:51871)
0929 03:11:14.913888 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:14.913996 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:14.915564 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:14.916262 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:14.916269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:14.916363 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:14.916878 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:14.916891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.917743 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.917747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:14.918766 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:14.918777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:14.919297 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:14.919302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:14.919479 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:14.920126 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:11:14.920128 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:14.920172 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:11:14.920240 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:11:14.920568 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:11:14.920580 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:11:14.920590 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 03:11:14.920721 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 03:11:14.921035 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":165,"threads_started":1}
I20260929 03:11:14.923151 30755 heartbeater.cc:347] Connected to a master server at 127.20.174.126:34353
I20260929 03:11:14.923417 30755 heartbeater.cc:464] Registering TS with master...
I20260929 03:11:14.924041 30755 heartbeater.cc:511] Master 127.20.174.126:34353 requested a full tablet report, sending...
I20260929 03:11:14.925257 30436 ts_manager.cc:194] Registered new tserver with Master: 5281e6fa2e44430b89bf1d332aa8302d (127.20.174.66:44383)
I20260929 03:11:14.926015 30436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.66:51871
W20260929 03:11:15.036059 30758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:15.036336 30758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:15.036374 30758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:15.040589 30758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:15.040648 30758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:15.040704 30758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:11:15.045471 30758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:15.046520 30758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:15.047912 30758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:15.058040 30766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:15.058174 30764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:15.058040 30763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:15.058450 30758 server_base.cc:1034] running on GCE node
I20260929 03:11:15.058892 30758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:15.059602 30758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:15.060827 30758 hybrid_clock.cc:648] HybridClock initialized: now 1790651475060784 us; error 57 us; skew 500 ppm
I20260929 03:11:15.068960 30758 webserver.cc:460] Webserver started at http://127.20.174.67:42225/ using document root <none> and password file <none>
I20260929 03:11:15.069643 30758 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:15.069747 30758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:15.070010 30758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:15.072508 30758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "69b9e2538c9e4f6e957c9533dafb632b"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "4e31507be4ffcea52c589f48091b7958"
server_key_iv: "1c091bc1f333216f85703eef4db4b527"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.073144 30758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "69b9e2538c9e4f6e957c9533dafb632b"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "4e31507be4ffcea52c589f48091b7958"
server_key_iv: "1c091bc1f333216f85703eef4db4b527"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.077260 30758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:11:15.080178 30772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.081614 30758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:11:15.081794 30758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "69b9e2538c9e4f6e957c9533dafb632b"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "4e31507be4ffcea52c589f48091b7958"
server_key_iv: "1c091bc1f333216f85703eef4db4b527"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.081948 30758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:15.108934 30758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:15.109687 30758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:15.109900 30758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:15.110594 30758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:11:15.111634 30758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:11:15.111707 30758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.111768 30758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:11:15.111790 30758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.120913 30758 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:32787
I20260929 03:11:15.120934 30885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:32787 every 8 connection(s)
I20260929 03:11:15.122006 30758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:11:15.131484 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:15.124032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:57415 (local address 127.20.174.126:34353)
0929 03:11:15.124182 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:15.124186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:15.125257 (+ 1071us) server_negotiation.cc:407] Connection header received
0929 03:11:15.126707 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:15.126710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:15.126759 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:15.126843 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:15.127768 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:15.128324 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:15.129499 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:15.129740 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:15.129853 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:15.130971 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:15.130974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:15.130976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:11:15.130988 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:15.131008 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:11:15.131017 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:15.131081 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:15.131238 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 03:11:15.131323 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:11:15.131477 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 30758
I20260929 03:11:15.131587 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:11:15.131886 21177 external_mini_cluster.cc:1428] Setting key 641b7a51ced5e48f0672b56223315372
I20260929 03:11:15.132476 30888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:15.124453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:34353 (local address 127.20.174.67:57415)
0929 03:11:15.125103 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:15.125143 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:15.126419 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:15.127004 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:15.127011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:15.127096 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:15.127617 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:15.127630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:15.128455 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:15.128458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:15.129359 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:15.129368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:15.129888 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:15.129891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:15.130033 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:15.130704 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:11:15.130706 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:15.130746 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:11:15.130813 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:11:15.131096 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:11:15.131102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:11:15.131107 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:11:15.131202 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:11:15.131430 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
I20260929 03:11:15.133694 30886 heartbeater.cc:347] Connected to a master server at 127.20.174.126:34353
I20260929 03:11:15.133946 30886 heartbeater.cc:464] Registering TS with master...
I20260929 03:11:15.134526 30886 heartbeater.cc:511] Master 127.20.174.126:34353 requested a full tablet report, sending...
I20260929 03:11:15.135577 30436 ts_manager.cc:194] Registered new tserver with Master: 69b9e2538c9e4f6e957c9533dafb632b (127.20.174.67:32787)
I20260929 03:11:15.136107 30436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.67:57415
I20260929 03:11:15.145905 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:11:15.148917 30475 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:15.147953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.20.174.126:34353)
0929 03:11:15.148121 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:15.148125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:15.148342 (+ 217us) server_negotiation.cc:407] Connection header received
0929 03:11:15.148448 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:15.148450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:15.148490 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:15.148560 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:15.148764 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33164: BlockingRecv error: recv got EOF from 127.0.0.1:33164 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:11:15.149426 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 30496
I20260929 03:11:15.155435 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 30627
I20260929 03:11:15.160742 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 30758
I20260929 03:11:15.166426 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 30405
2026-09-29T03:11:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (943 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T03:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:11:15Z Disabled control of system clock
I20260929 03:11:15.196764 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:41601
--webserver_interface=127.20.174.126
--webserver_port=0
--builtin_ntp_servers=127.20.174.84:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:41601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:11:15.315685 30897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:15.315994 30897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:15.316033 30897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:15.320133 30897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:11:15.320200 30897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:15.320216 30897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:15.320233 30897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:11:15.320266 30897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:11:15.325388 30897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:41601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:41601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:15.326512 30897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:15.328068 30897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:15.336948 30902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:15.337122 30897 server_base.cc:1034] running on GCE node
W20260929 03:11:15.336987 30905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:15.336990 30903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:15.338151 30897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:15.339505 30897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:15.340688 30897 hybrid_clock.cc:648] HybridClock initialized: now 1790651475340675 us; error 30 us; skew 500 ppm
I20260929 03:11:15.342893 30897 webserver.cc:460] Webserver started at http://127.20.174.126:40339/ using document root <none> and password file <none>
I20260929 03:11:15.343492 30897 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:15.343559 30897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:15.343827 30897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:15.346030 30897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "797035fca09a4adcb0bfbb8ff9cb03de"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "b6f274db904c1e7a8aa66a51a50b8fa2"
server_key_iv: "cc631be667fc81f9e57095bf75820486"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.346693 30897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "797035fca09a4adcb0bfbb8ff9cb03de"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "b6f274db904c1e7a8aa66a51a50b8fa2"
server_key_iv: "cc631be667fc81f9e57095bf75820486"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.350696 30897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:11:15.353345 30911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.354823 30897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:11:15.354962 30897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "797035fca09a4adcb0bfbb8ff9cb03de"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "b6f274db904c1e7a8aa66a51a50b8fa2"
server_key_iv: "cc631be667fc81f9e57095bf75820486"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.355082 30897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:15.359862 30897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:15.360505 30897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:15.360698 30897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:15.367900 30897 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:41601
I20260929 03:11:15.367911 30963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:41601 every 8 connection(s)
I20260929 03:11:15.368948 30897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:11:15.372378 30964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:11:15.372905 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 30897
I20260929 03:11:15.373039 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:11:15.373344 21177 external_mini_cluster.cc:1428] Setting key 9cd85ef1ba663450a08c407b8f21a588
I20260929 03:11:15.380074 30964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de: Bootstrap starting.
I20260929 03:11:15.381011 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:15.375212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.20.174.126:41601)
0929 03:11:15.375927 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:15.375945 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:15.375992 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:11:15.376854 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:15.376881 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:15.377368 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:15.377853 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:15.379335 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:15.379342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:15.379359 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:11:15.379372 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:15.379471 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:11:15.379497 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:15.379643 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:15.379820 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:11:15.380080 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":111,"threads_started":1}
I20260929 03:11:15.383466 30964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de: Neither blocks nor log segments found. Creating new log.
I20260929 03:11:15.384464 30964 log.cc:824] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de: Log is configured to *not* fsync() on all Append() calls
I20260929 03:11:15.386680 30964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de: No bootstrap required, opened a new log
I20260929 03:11:15.392179 30964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797035fca09a4adcb0bfbb8ff9cb03de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41601 } }
I20260929 03:11:15.392521 30964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:11:15.392596 30964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797035fca09a4adcb0bfbb8ff9cb03de, State: Initialized, Role: FOLLOWER
I20260929 03:11:15.393180 30964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797035fca09a4adcb0bfbb8ff9cb03de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41601 } }
I20260929 03:11:15.393365 30964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:11:15.393437 30964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:11:15.393554 30964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:11:15.394949 30964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797035fca09a4adcb0bfbb8ff9cb03de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41601 } }
I20260929 03:11:15.395350 30964 leader_election.cc:302] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 797035fca09a4adcb0bfbb8ff9cb03de; no voters:
I20260929 03:11:15.395720 30964 leader_election.cc:288] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:11:15.396015 30969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:11:15.396215 30969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [term 1 LEADER]: Becoming Leader. State: Replica: 797035fca09a4adcb0bfbb8ff9cb03de, State: Running, Role: LEADER
I20260929 03:11:15.396590 30969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797035fca09a4adcb0bfbb8ff9cb03de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41601 } }
I20260929 03:11:15.397200 30964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:11:15.398535 30970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "797035fca09a4adcb0bfbb8ff9cb03de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797035fca09a4adcb0bfbb8ff9cb03de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41601 } } }
I20260929 03:11:15.399240 30970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [sys.catalog]: This master's current role is: LEADER
I20260929 03:11:15.399935 30971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 797035fca09a4adcb0bfbb8ff9cb03de. Latest consensus state: current_term: 1 leader_uuid: "797035fca09a4adcb0bfbb8ff9cb03de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797035fca09a4adcb0bfbb8ff9cb03de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 41601 } } }
I20260929 03:11:15.400234 30971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de [sys.catalog]: This master's current role is: LEADER
W20260929 03:11:15.402782 30983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:11:15.402856 30983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:11:15.402930 30984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:11:15.403896 30984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:11:15.412770 30984 catalog_manager.cc:1347] Generated new cluster ID: 61ed7ef02d6e4a19a5f5c1ccd7c4f21e
I20260929 03:11:15.412858 30984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:11:15.430091 30984 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:11:15.430523 30984 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:11:15.435978 30984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 797035fca09a4adcb0bfbb8ff9cb03de: Generated new TSK 0
I20260929 03:11:15.436828 30984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:11:15.441699 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:41601
--builtin_ntp_servers=127.20.174.84:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:11:15.560475 30988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:15.560755 30988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:15.560838 30988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:15.565618 30988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:15.565701 30988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:15.565790 30988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:11:15.571547 30988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:15.572842 30988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:15.574292 30988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:15.584627 30996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:15.584643 30994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:15.584832 30993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:15.584975 30988 server_base.cc:1034] running on GCE node
I20260929 03:11:15.585413 30988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:15.586274 30988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:15.587458 30988 hybrid_clock.cc:648] HybridClock initialized: now 1790651475587442 us; error 34 us; skew 500 ppm
I20260929 03:11:15.589627 30988 webserver.cc:460] Webserver started at http://127.20.174.65:36731/ using document root <none> and password file <none>
I20260929 03:11:15.590190 30988 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:15.590251 30988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:15.590425 30988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:15.592592 30988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "0796ca9e4cb445a9820bcb7486ffe02c"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "c7790418fb5ca5bd90a8bff2077f73ba"
server_key_iv: "5a622cf6e3483508779e4aacfe4021e6"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.593163 30988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "0796ca9e4cb445a9820bcb7486ffe02c"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "c7790418fb5ca5bd90a8bff2077f73ba"
server_key_iv: "5a622cf6e3483508779e4aacfe4021e6"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.597081 30988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 03:11:15.600100 31002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.601400 30988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:11:15.601538 30988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "0796ca9e4cb445a9820bcb7486ffe02c"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "c7790418fb5ca5bd90a8bff2077f73ba"
server_key_iv: "5a622cf6e3483508779e4aacfe4021e6"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.601657 30988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:15.606158 30988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:15.606829 30988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:15.607024 30988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:15.607570 30988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:11:15.608528 30988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:11:15.608577 30988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.608644 30988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:11:15.608675 30988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.617668 30988 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:46765
I20260929 03:11:15.617749 31115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:46765 every 8 connection(s)
I20260929 03:11:15.618717 30988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:11:15.624922 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:15.620633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:34069 (local address 127.20.174.126:41601)
0929 03:11:15.620771 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:15.620775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:15.621830 (+ 1055us) server_negotiation.cc:407] Connection header received
0929 03:11:15.623275 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:15.623279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:15.623320 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:15.623409 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:15.624520 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:15.624523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:15.624525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:11:15.624538 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:15.624559 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:11:15.624568 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:15.624650 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:15.624770 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 03:11:15.624811 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:11:15.625902 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:15.620996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:41601 (local address 127.20.174.65:34069)
0929 03:11:15.621654 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:15.621696 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:15.623035 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:15.623580 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:15.623588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:15.623681 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:15.624246 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:11:15.624248 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:15.624290 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:11:15.624363 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:11:15.624646 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:11:15.624652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:11:15.624657 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:11:15.624794 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 03:11:15.625097 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260929 03:11:15.627338 31116 heartbeater.cc:347] Connected to a master server at 127.20.174.126:41601
I20260929 03:11:15.627625 31116 heartbeater.cc:464] Registering TS with master...
I20260929 03:11:15.628101 31116 heartbeater.cc:511] Master 127.20.174.126:41601 requested a full tablet report, sending...
I20260929 03:11:15.628646 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 30988
I20260929 03:11:15.628787 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:11:15.629136 21177 external_mini_cluster.cc:1428] Setting key ed532e32d1768f97ba8295d82d555990
I20260929 03:11:15.629963 30928 ts_manager.cc:194] Registered new tserver with Master: 0796ca9e4cb445a9820bcb7486ffe02c (127.20.174.65:46765)
I20260929 03:11:15.631582 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:41601
--builtin_ntp_servers=127.20.174.84:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:11:15.753859 31119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:15.754146 31119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:15.754186 31119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:15.758476 31119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:15.758579 31119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:15.758698 31119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:11:15.763695 31119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:15.764779 31119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:15.766234 31119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:15.776506 31127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:15.776535 31125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:15.776510 31124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:15.776562 31119 server_base.cc:1034] running on GCE node
I20260929 03:11:15.777243 31119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:15.778062 31119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:15.779280 31119 hybrid_clock.cc:648] HybridClock initialized: now 1790651475779230 us; error 76 us; skew 500 ppm
I20260929 03:11:15.781494 31119 webserver.cc:460] Webserver started at http://127.20.174.66:43913/ using document root <none> and password file <none>
I20260929 03:11:15.782128 31119 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:15.782194 31119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:15.782541 31119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:15.784749 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "6fe4aa97d91547caac3b3bb6dd3e3780"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "41fcc31d5f447a8cd3feb9bf25ad75d0"
server_key_iv: "aeb83b8952efd31a0407df05f1eeec4c"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.785344 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "6fe4aa97d91547caac3b3bb6dd3e3780"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "41fcc31d5f447a8cd3feb9bf25ad75d0"
server_key_iv: "aeb83b8952efd31a0407df05f1eeec4c"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.789408 31119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:11:15.792188 31133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.793533 31119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:11:15.793672 31119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "6fe4aa97d91547caac3b3bb6dd3e3780"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "41fcc31d5f447a8cd3feb9bf25ad75d0"
server_key_iv: "aeb83b8952efd31a0407df05f1eeec4c"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.793866 31119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:15.799021 31119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:15.799664 31119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:15.799860 31119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:15.800431 31119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:11:15.801370 31119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:11:15.801419 31119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.801484 31119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:11:15.801525 31119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.811187 31119 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:37959
I20260929 03:11:15.811275 31246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:37959 every 8 connection(s)
I20260929 03:11:15.812161 31119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:11:15.818594 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 31119
I20260929 03:11:15.818780 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:11:15.819056 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:15.814187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:55471 (local address 127.20.174.126:41601)
0929 03:11:15.814382 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:15.814387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:15.815541 (+ 1154us) server_negotiation.cc:407] Connection header received
0929 03:11:15.816864 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:15.816867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:15.816909 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:15.816989 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:15.818376 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:15.818382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:15.818389 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:11:15.818407 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:15.818458 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:11:15.818472 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:15.818600 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:15.818740 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 03:11:15.818892 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 03:11:15.819125 21177 external_mini_cluster.cc:1428] Setting key 6bd6e937756e50a6f9d493950f875ffa
I20260929 03:11:15.819999 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:15.814686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:41601 (local address 127.20.174.66:55471)
0929 03:11:15.815374 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:15.815417 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:15.816652 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:15.817157 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:15.817164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:15.817254 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:15.817885 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:11:15.817886 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:15.817929 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:11:15.818041 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:11:15.818570 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:11:15.818577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:11:15.818582 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:11:15.818769 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 03:11:15.819012 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
I20260929 03:11:15.821341 31247 heartbeater.cc:347] Connected to a master server at 127.20.174.126:41601
I20260929 03:11:15.821522 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:41601
--builtin_ntp_servers=127.20.174.84:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 03:11:15.821604 31247 heartbeater.cc:464] Registering TS with master...
I20260929 03:11:15.822048 31247 heartbeater.cc:511] Master 127.20.174.126:41601 requested a full tablet report, sending...
I20260929 03:11:15.823269 30928 ts_manager.cc:194] Registered new tserver with Master: 6fe4aa97d91547caac3b3bb6dd3e3780 (127.20.174.66:37959)
W20260929 03:11:15.937820 31250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:15.938097 31250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:15.938140 31250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:15.942489 31250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:15.942580 31250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:15.942680 31250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:11:15.947736 31250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:15.948910 31250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:15.950385 31250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:15.960773 31258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:15.960778 31255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:15.960844 31250 server_base.cc:1034] running on GCE node
W20260929 03:11:15.960778 31256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:15.961429 31250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:15.962145 31250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:15.963308 31250 hybrid_clock.cc:648] HybridClock initialized: now 1790651475963272 us; error 58 us; skew 500 ppm
I20260929 03:11:15.965299 31250 webserver.cc:460] Webserver started at http://127.20.174.67:44435/ using document root <none> and password file <none>
I20260929 03:11:15.965934 31250 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:15.966044 31250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:15.966284 31250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:15.968542 31250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "36d36d00a46a426da3bbf9736e727d0b"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "14b1cac8ee3c3dbab767e787e61e85c8"
server_key_iv: "387cbe6b9bd564df76b4460ffd271ae8"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.969146 31250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "36d36d00a46a426da3bbf9736e727d0b"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "14b1cac8ee3c3dbab767e787e61e85c8"
server_key_iv: "387cbe6b9bd564df76b4460ffd271ae8"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.973310 31250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:11:15.976301 31264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.977712 31250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:11:15.977867 31250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "36d36d00a46a426da3bbf9736e727d0b"
format_stamp: "Formatted at 2026-09-29 03:11:15 on dist-test-slave-k1dg"
server_key: "14b1cac8ee3c3dbab767e787e61e85c8"
server_key_iv: "387cbe6b9bd564df76b4460ffd271ae8"
server_key_version: "encryptionkey@0"
I20260929 03:11:15.978010 31250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:15.983131 31250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:15.983829 31250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:15.984037 31250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:15.984619 31250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:11:15.985603 31250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:11:15.985678 31250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.985750 31250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:11:15.985786 31250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:15.995507 31250 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:42901
I20260929 03:11:15.995568 31377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:42901 every 8 connection(s)
I20260929 03:11:15.996537 31250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:11:15.998695 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 31250
I20260929 03:11:15.998831 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:11:15.999125 21177 external_mini_cluster.cc:1428] Setting key 3e9be0e2c41617909d4dcdadcc34afe2
I20260929 03:11:16.003669 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:15.998882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:42743 (local address 127.20.174.126:41601)
0929 03:11:15.999029 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:15.999034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:16.000427 (+ 1393us) server_negotiation.cc:407] Connection header received
0929 03:11:16.001990 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:16.001993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:16.002034 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:16.002113 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:16.003237 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:16.003240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:16.003242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:11:16.003260 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:16.003279 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:11:16.003288 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:16.003363 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:16.003521 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:11:16.003560 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:11:16.004585 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:15.999466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:41601 (local address 127.20.174.67:42743)
0929 03:11:16.000260 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:16.000316 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:16.001784 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:16.002254 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:16.002261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:16.002353 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:16.002980 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:11:16.002982 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:16.003026 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:11:16.003097 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:11:16.003367 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:11:16.003373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:11:16.003378 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:11:16.003545 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 03:11:16.003749 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":155,"threads_started":1}
I20260929 03:11:16.005738 31378 heartbeater.cc:347] Connected to a master server at 127.20.174.126:41601
I20260929 03:11:16.006037 31378 heartbeater.cc:464] Registering TS with master...
I20260929 03:11:16.006599 31378 heartbeater.cc:511] Master 127.20.174.126:41601 requested a full tablet report, sending...
I20260929 03:11:16.007717 30928 ts_manager.cc:194] Registered new tserver with Master: 36d36d00a46a426da3bbf9736e727d0b (127.20.174.67:42901)
I20260929 03:11:16.013386 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:11:16.016642 30967 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:11:16.015508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.20.174.126:41601)
0929 03:11:16.015673 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:16.015678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:16.015956 (+ 278us) server_negotiation.cc:407] Connection header received
0929 03:11:16.016123 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:16.016126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:16.016171 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:16.016259 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:16.016505 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59514: BlockingRecv error: recv got EOF from 127.0.0.1:59514 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:11:16.017139 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 30988
I20260929 03:11:16.022677 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 31119
I20260929 03:11:16.028226 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 31250
I20260929 03:11:16.035665 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 30897
2026-09-29T03:11:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (868 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:11:16 dist-test-slave-k1dg krb5kdc[31388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:11:16 dist-test-slave-k1dg krb5kdc[31388](info): set up 2 sockets
Sep 29 03:11:16 dist-test-slave-k1dg krb5kdc[31388](info): commencing operation
krb5kdc: starting...
W20260929 03:11:17.075647 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:11:17 dist-test-slave-k1dg krb5kdc[31388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:11:17.206640 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:33741
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:33741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:11:17.323325 31401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:17.323625 31401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:17.323669 31401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:17.328070 31401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:11:17.328152 31401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:17.328172 31401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:17.328190 31401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:11:17.328207 31401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:11:17.334072 31401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:33741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:33741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:17.335368 31401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:17.336486 31401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:17.345891 31407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:17.345943 31406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:17.345934 31409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:17.346189 31401 server_base.cc:1034] running on GCE node
I20260929 03:11:17.346892 31401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:17.348152 31401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:17.349357 31401 hybrid_clock.cc:648] HybridClock initialized: now 1790651477349343 us; error 30 us; skew 500 ppm
Sep 29 03:11:17 dist-test-slave-k1dg krb5kdc[31388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651477, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:11:17.358314 31401 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:11:17.359511 31401 webserver.cc:460] Webserver started at http://127.20.174.126:38523/ using document root <none> and password file <none>
I20260929 03:11:17.360188 31401 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:17.360307 31401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:17.360592 31401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:17.363152 31401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "5d69824dc03040b48b3fc016c0f27306"
format_stamp: "Formatted at 2026-09-29 03:11:17 on dist-test-slave-k1dg"
server_key: "4354abd94e3b1ebe306ea3d85df2c0ef"
server_key_iv: "d4ac901482cba52194d048edf85b14ed"
server_key_version: "encryptionkey@0"
I20260929 03:11:17.363832 31401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "5d69824dc03040b48b3fc016c0f27306"
format_stamp: "Formatted at 2026-09-29 03:11:17 on dist-test-slave-k1dg"
server_key: "4354abd94e3b1ebe306ea3d85df2c0ef"
server_key_iv: "d4ac901482cba52194d048edf85b14ed"
server_key_version: "encryptionkey@0"
I20260929 03:11:17.368417 31401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:11:17.371472 31416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:17.372960 31401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:11:17.373153 31401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "5d69824dc03040b48b3fc016c0f27306"
format_stamp: "Formatted at 2026-09-29 03:11:17 on dist-test-slave-k1dg"
server_key: "4354abd94e3b1ebe306ea3d85df2c0ef"
server_key_iv: "d4ac901482cba52194d048edf85b14ed"
server_key_version: "encryptionkey@0"
I20260929 03:11:17.373293 31401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:17.387157 31401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:17.390723 31401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:17.390967 31401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:17.398649 31401 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:33741
I20260929 03:11:17.398927 31468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:33741 every 8 connection(s)
I20260929 03:11:17.399765 31401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:11:17.402665 31469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:11:17.403623 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 31401
I20260929 03:11:17.403775 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:11:17.404076 21177 external_mini_cluster.cc:1428] Setting key 697e81f3641134941a4489f277d8eac5
I20260929 03:11:17.409669 31469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306: Bootstrap starting.
I20260929 03:11:17.412405 31469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306: Neither blocks nor log segments found. Creating new log.
I20260929 03:11:17.413177 31469 log.cc:824] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:11:17 dist-test-slave-k1dg krb5kdc[31388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:11:17.415551 31469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306: No bootstrap required, opened a new log
I20260929 03:11:17.419664 31469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69824dc03040b48b3fc016c0f27306" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33741 } }
I20260929 03:11:17.419896 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:17.405795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.20.174.126:33741)
0929 03:11:17.406253 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:17.406273 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:17.406317 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:11:17.407178 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:17.407203 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:17.407636 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:17.408045 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:17.408789 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:17.410204 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:17.411191 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:17.411873 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:17.412043 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:17.414899 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:17.414904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:17.414918 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:11:17.414948 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:17.416820 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:11:17.417327 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:11:17.417335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:11:17.417346 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:11:17.417491 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:11:17.417800 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:11:17.417804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:11:17.417806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:11:17.418246 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:17.418517 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:17.418764 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 03:11:17.419090 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":125,"threads_started":1}
I20260929 03:11:17.419962 31469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:11:17.420034 31469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d69824dc03040b48b3fc016c0f27306, State: Initialized, Role: FOLLOWER
I20260929 03:11:17.420604 31469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69824dc03040b48b3fc016c0f27306" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33741 } }
I20260929 03:11:17.420791 31469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:11:17.420850 31469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:11:17.420946 31469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:11:17.422048 31469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69824dc03040b48b3fc016c0f27306" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33741 } }
I20260929 03:11:17.422514 31469 leader_election.cc:302] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d69824dc03040b48b3fc016c0f27306; no voters:
I20260929 03:11:17.422930 31469 leader_election.cc:288] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:11:17.423012 31474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:11:17.423249 31474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [term 1 LEADER]: Becoming Leader. State: Replica: 5d69824dc03040b48b3fc016c0f27306, State: Running, Role: LEADER
I20260929 03:11:17.423583 31474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69824dc03040b48b3fc016c0f27306" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33741 } }
I20260929 03:11:17.424371 31469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:11:17.425168 31475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d69824dc03040b48b3fc016c0f27306. Latest consensus state: current_term: 1 leader_uuid: "5d69824dc03040b48b3fc016c0f27306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69824dc03040b48b3fc016c0f27306" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33741 } } }
I20260929 03:11:17.425341 31475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [sys.catalog]: This master's current role is: LEADER
I20260929 03:11:17.425550 31476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d69824dc03040b48b3fc016c0f27306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69824dc03040b48b3fc016c0f27306" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33741 } } }
I20260929 03:11:17.425665 31476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306 [sys.catalog]: This master's current role is: LEADER
I20260929 03:11:17.425833 31481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:11:17.429574 31481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:11:17.436321 31481 catalog_manager.cc:1347] Generated new cluster ID: 90a08d91d30b468192a213f117a4bddb
I20260929 03:11:17.436414 31481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:11:17.450855 31481 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:11:17.451932 31481 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:11:17.463115 31481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306: Generated new TSK 0
I20260929 03:11:17.463925 31481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:11:17.498826 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33741
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:11:17.624094 31495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:17.624450 31495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:17.624505 31495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:17.629074 31495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:17.629153 31495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:17.629215 31495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:11:17.634832 31495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:17.635937 31495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:17.636965 31495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:17.647413 31503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:17.647436 31500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:17.647498 31501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:17.647728 31495 server_base.cc:1034] running on GCE node
I20260929 03:11:17.648315 31495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:17.648985 31495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:17.650199 31495 hybrid_clock.cc:648] HybridClock initialized: now 1790651477650182 us; error 29 us; skew 500 ppm
Sep 29 03:11:17 dist-test-slave-k1dg krb5kdc[31388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651477, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:11:17.653218 31495 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:11:17.654199 31495 webserver.cc:460] Webserver started at http://127.20.174.65:43603/ using document root <none> and password file <none>
I20260929 03:11:17.654860 31495 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:17.654958 31495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:17.655195 31495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:17.657402 31495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "737042732c374706bdd28e170967122e"
format_stamp: "Formatted at 2026-09-29 03:11:17 on dist-test-slave-k1dg"
server_key: "e0d1b5baf954bd030db3e14d0e501bed"
server_key_iv: "48cea1d197de4b97c49703f7b2641aa2"
server_key_version: "encryptionkey@0"
I20260929 03:11:17.658015 31495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "737042732c374706bdd28e170967122e"
format_stamp: "Formatted at 2026-09-29 03:11:17 on dist-test-slave-k1dg"
server_key: "e0d1b5baf954bd030db3e14d0e501bed"
server_key_iv: "48cea1d197de4b97c49703f7b2641aa2"
server_key_version: "encryptionkey@0"
I20260929 03:11:17.661969 31495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:11:17.664671 31510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:17.666121 31495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:11:17.666280 31495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "737042732c374706bdd28e170967122e"
format_stamp: "Formatted at 2026-09-29 03:11:17 on dist-test-slave-k1dg"
server_key: "e0d1b5baf954bd030db3e14d0e501bed"
server_key_iv: "48cea1d197de4b97c49703f7b2641aa2"
server_key_version: "encryptionkey@0"
I20260929 03:11:17.666405 31495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:17.692328 31495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:17.695870 31495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:17.696102 31495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:17.696760 31495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:11:17.697698 31495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:11:17.697782 31495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:17.697871 31495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:11:17.697930 31495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:17.708740 31495 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:32897
I20260929 03:11:17.708761 31623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:32897 every 8 connection(s)
I20260929 03:11:17.709793 31495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:11:17.717245 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 31495
I20260929 03:11:17.717381 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:11:17.717707 21177 external_mini_cluster.cc:1428] Setting key cafb9f90d37e97292799cb67247a31c7
Sep 29 03:11:17 dist-test-slave-k1dg krb5kdc[31388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651477, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:11:17.727587 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:17.711753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:47535 (local address 127.20.174.126:33741)
0929 03:11:17.711906 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:17.711910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:17.712894 (+ 984us) server_negotiation.cc:407] Connection header received
0929 03:11:17.714347 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:17.714350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:17.714404 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:17.714581 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:17.715988 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:17.716630 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:17.718377 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:17.718988 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:17.719243 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:17.722727 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:17.722734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:17.722741 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:11:17.722778 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:17.724740 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:11:17.725487 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:11:17.725493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:11:17.725498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:11:17.725602 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:11:17.726176 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:11:17.726182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:11:17.726184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:11:17.726472 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:17.726582 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:17.727211 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 03:11:17.727369 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:11:17.728461 31626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:17.712213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:33741 (local address 127.20.174.65:47535)
0929 03:11:17.712717 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:17.712763 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:17.714097 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:17.714741 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:17.714748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:17.715135 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:17.715667 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:17.715681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:17.716765 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:17.716769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:17.718227 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:17.718238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:17.719326 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:17.719331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:17.719509 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:17.720410 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:17.720438 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:17.722512 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:11:17.724870 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:11:17.724875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:11:17.724889 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:11:17.725379 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:11:17.725855 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:11:17.725860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:11:17.725863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:11:17.726044 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:11:17.726595 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:11:17.726601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:11:17.726941 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 03:11:17.727122 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 03:11:17.727420 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:11:17.729984 31624 heartbeater.cc:347] Connected to a master server at 127.20.174.126:33741
I20260929 03:11:17.730228 31624 heartbeater.cc:464] Registering TS with master...
I20260929 03:11:17.730834 31624 heartbeater.cc:511] Master 127.20.174.126:33741 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:11:17.732606 31433 ts_manager.cc:194] Registered new tserver with Master: 737042732c374706bdd28e170967122e (127.20.174.65:32897)
I20260929 03:11:17.733873 31433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:47535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:11:17.750692 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33741
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:11:17.867391 31629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:17.867671 31629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:17.867738 31629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:17.872176 31629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:17.872236 31629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:17.872323 31629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:11:17.880685 31629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:17.882091 31629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:17.883426 31629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:17.893961 31634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:17.893961 31635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:17.893999 31637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:17.894265 31629 server_base.cc:1034] running on GCE node
I20260929 03:11:17.894799 31629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:17.895424 31629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:17.896591 31629 hybrid_clock.cc:648] HybridClock initialized: now 1790651477896576 us; error 37 us; skew 500 ppm
Sep 29 03:11:17 dist-test-slave-k1dg krb5kdc[31388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651477, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:11:17.899538 31629 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:11:17.900454 31629 webserver.cc:460] Webserver started at http://127.20.174.66:39293/ using document root <none> and password file <none>
I20260929 03:11:17.901023 31629 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:17.901084 31629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:17.901309 31629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:17.903458 31629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "4b9186f09cd0413c8aca153014a035ed"
format_stamp: "Formatted at 2026-09-29 03:11:17 on dist-test-slave-k1dg"
server_key: "d0d1ef4ef1530ffd7d82ce0eb454dbe5"
server_key_iv: "6d5f07d9f218e48fa2fadad36ddd302b"
server_key_version: "encryptionkey@0"
I20260929 03:11:17.904029 31629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "4b9186f09cd0413c8aca153014a035ed"
format_stamp: "Formatted at 2026-09-29 03:11:17 on dist-test-slave-k1dg"
server_key: "d0d1ef4ef1530ffd7d82ce0eb454dbe5"
server_key_iv: "6d5f07d9f218e48fa2fadad36ddd302b"
server_key_version: "encryptionkey@0"
I20260929 03:11:17.908044 31629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:11:17.910744 31644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:17.912070 31629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:11:17.912292 31629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "4b9186f09cd0413c8aca153014a035ed"
format_stamp: "Formatted at 2026-09-29 03:11:17 on dist-test-slave-k1dg"
server_key: "d0d1ef4ef1530ffd7d82ce0eb454dbe5"
server_key_iv: "6d5f07d9f218e48fa2fadad36ddd302b"
server_key_version: "encryptionkey@0"
I20260929 03:11:17.912412 31629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:17.926718 31629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:17.929940 31629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:17.930171 31629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:17.930872 31629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:11:17.931780 31629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:11:17.931859 31629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:17.931932 31629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:11:17.931986 31629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:17.941457 31629 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:44969
I20260929 03:11:17.941471 31757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:44969 every 8 connection(s)
I20260929 03:11:17.942414 31629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:11:17.947854 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 31629
I20260929 03:11:17.948048 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:11:17.948342 21177 external_mini_cluster.cc:1428] Setting key fafbc564db7925d757a8e4249e7ef1cf
Sep 29 03:11:17 dist-test-slave-k1dg krb5kdc[31388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651477, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:11:17.958228 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:17.944511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:43371 (local address 127.20.174.126:33741)
0929 03:11:17.944670 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:17.944674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:17.945714 (+ 1040us) server_negotiation.cc:407] Connection header received
0929 03:11:17.947308 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:17.947311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:17.947374 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:17.947489 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:17.949013 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:17.949555 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:17.950869 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:17.951093 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:17.951207 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:17.953861 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:17.953867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:17.953874 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:11:17.953912 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:17.955765 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:11:17.956391 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:11:17.956397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:11:17.956399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:11:17.956500 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:11:17.956884 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:11:17.956889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:11:17.956893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:11:17.957161 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:17.957275 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:17.957870 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 03:11:17.958037 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:11:17.959122 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:17.944967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:33741 (local address 127.20.174.66:43371)
0929 03:11:17.945461 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:17.945506 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:17.947074 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:17.947683 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:17.947691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:17.948208 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:17.948841 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:17.948863 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:17.949693 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:17.949696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:17.950728 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:17.950739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:17.951250 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:17.951254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:17.951439 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:17.952087 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:17.952104 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:17.953613 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:11:17.955976 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:11:17.955981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:11:17.955996 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:11:17.956274 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:11:17.956619 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:11:17.956621 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:11:17.956623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:11:17.956778 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:11:17.957306 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:11:17.957312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:11:17.957564 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 03:11:17.957764 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 03:11:17.958130 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":112,"threads_started":1}
I20260929 03:11:17.960600 31758 heartbeater.cc:347] Connected to a master server at 127.20.174.126:33741
I20260929 03:11:17.960897 31758 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:11:17.961547 31758 heartbeater.cc:511] Master 127.20.174.126:33741 requested a full tablet report, sending...
I20260929 03:11:17.962934 31433 ts_manager.cc:194] Registered new tserver with Master: 4b9186f09cd0413c8aca153014a035ed (127.20.174.66:44969)
I20260929 03:11:17.963654 31433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:43371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:11:17.978221 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33741
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:11:18.095924 31763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:11:18.096220 31763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:11:18.096264 31763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:11:18.100425 31763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:11:18.100492 31763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:11:18.100602 31763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:11:18.105957 31763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:11:18.107000 31763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:11:18.108021 31763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:11:18.118008 31769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:11:18.118075 31771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:18.118151 31763 server_base.cc:1034] running on GCE node
W20260929 03:11:18.118009 31768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:11:18.118716 31763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:11:18.119364 31763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:11:18.120550 31763 hybrid_clock.cc:648] HybridClock initialized: now 1790651478120535 us; error 29 us; skew 500 ppm
Sep 29 03:11:18 dist-test-slave-k1dg krb5kdc[31388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651478, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:11:18.123427 31763 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:11:18.124420 31763 webserver.cc:460] Webserver started at http://127.20.174.67:46241/ using document root <none> and password file <none>
I20260929 03:11:18.125018 31763 fs_manager.cc:359] Metadata directory not provided
I20260929 03:11:18.125110 31763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:11:18.125373 31763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:11:18.127592 31763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "66cc85e37da544a895532038d1b1af65"
format_stamp: "Formatted at 2026-09-29 03:11:18 on dist-test-slave-k1dg"
server_key: "89913ba051d02dc9ba2b107abfc0d4a7"
server_key_iv: "11d762cb2f01e9a5e82c9346ffa11470"
server_key_version: "encryptionkey@0"
I20260929 03:11:18.128247 31763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "66cc85e37da544a895532038d1b1af65"
format_stamp: "Formatted at 2026-09-29 03:11:18 on dist-test-slave-k1dg"
server_key: "89913ba051d02dc9ba2b107abfc0d4a7"
server_key_iv: "11d762cb2f01e9a5e82c9346ffa11470"
server_key_version: "encryptionkey@0"
I20260929 03:11:18.132270 31763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:11:18.135140 31778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:18.136566 31763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:11:18.136725 31763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "66cc85e37da544a895532038d1b1af65"
format_stamp: "Formatted at 2026-09-29 03:11:18 on dist-test-slave-k1dg"
server_key: "89913ba051d02dc9ba2b107abfc0d4a7"
server_key_iv: "11d762cb2f01e9a5e82c9346ffa11470"
server_key_version: "encryptionkey@0"
I20260929 03:11:18.136857 31763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:11:18.147521 31763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:11:18.151172 31763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:11:18.151387 31763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:11:18.152019 31763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:11:18.152876 31763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:11:18.152952 31763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:18.153031 31763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:11:18.153076 31763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:11:18.162401 31763 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:32907
I20260929 03:11:18.162418 31891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:32907 every 8 connection(s)
I20260929 03:11:18.163340 31763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:11:18.165432 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 31763
I20260929 03:11:18.165575 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:11:18.165962 21177 external_mini_cluster.cc:1428] Setting key a3bb118a7bfa07e390013a5095eafe8d
Sep 29 03:11:18 dist-test-slave-k1dg krb5kdc[31388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651478, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:11:18.178707 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.165103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:59637 (local address 127.20.174.126:33741)
0929 03:11:18.165326 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:18.165332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:18.166224 (+ 892us) server_negotiation.cc:407] Connection header received
0929 03:11:18.168323 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:18.168326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:18.168369 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:18.168432 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:18.169678 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.170230 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.171715 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.171958 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.172055 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.174749 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:18.174752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:18.174755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:11:18.174783 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:18.176383 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:11:18.177017 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:11:18.177020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:11:18.177021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:11:18.177102 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:11:18.177483 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:11:18.177485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:11:18.177486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:11:18.177669 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:18.177747 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:18.178381 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 03:11:18.178553 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 03:11:18.179797 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.165478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:33741 (local address 127.20.174.67:59637)
0929 03:11:18.166048 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:18.166107 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:18.167986 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:18.168592 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:18.168599 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:18.168953 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:11:18.169536 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.169553 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.170368 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.170371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.171560 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.171572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.172138 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.172141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.172307 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.172906 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:11:18.172924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:11:18.174575 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:11:18.176509 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:11:18.176515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:11:18.176528 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:11:18.176913 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:11:18.177218 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:11:18.177220 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:11:18.177222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:11:18.177381 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:11:18.177788 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:11:18.177794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:11:18.178112 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 03:11:18.178311 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 03:11:18.178712 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":128,"threads_started":1}
I20260929 03:11:18.181016 31892 heartbeater.cc:347] Connected to a master server at 127.20.174.126:33741
I20260929 03:11:18.181282 31892 heartbeater.cc:464] Registering TS with master...
I20260929 03:11:18.181818 31892 heartbeater.cc:511] Master 127.20.174.126:33741 requested a full tablet report, sending...
I20260929 03:11:18.183007 31433 ts_manager.cc:194] Registered new tserver with Master: 66cc85e37da544a895532038d1b1af65 (127.20.174.67:32907)
I20260929 03:11:18.183655 31433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:59637
I20260929 03:11:18.192350 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:11:18.202005 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.194377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.20.174.126:33741)
0929 03:11:18.194543 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:18.194547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:18.194650 (+ 103us) server_negotiation.cc:407] Connection header received
0929 03:11:18.194761 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:18.194765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:18.194815 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:18.194895 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:11:18.195627 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.196180 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.197108 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.197384 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.197477 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.199031 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:11:18.199034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:11:18.199036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:11:18.199060 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:11:18.200417 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:11:18.200845 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:11:18.200848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:11:18.200849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:11:18.200934 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:11:18.201179 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:11:18.201181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:11:18.201182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:11:18.201442 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:11:18.201563 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:18.201777 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 03:11:18.201882 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 03:11:18.205166 31433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:11:18.208388 31433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:11:18.224289 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.219064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.20.174.65:32897)
0929 03:11:18.219369 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:18.219384 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:18.219407 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:11:18.219491 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:18.219498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:18.219680 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:18.219830 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:11:18.220067 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.221226 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.222977 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.223583 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.223724 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.223887 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:18.224028 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 03:11:18.224124 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":112,"threads_started":1}
I20260929 03:11:18.224278 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.218612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:32897 (local address 127.0.0.1:37978)
0929 03:11:18.218869 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:18.218890 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:18.219125 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:18.219833 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:18.219836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:18.219854 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:11:18.219974 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.219981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.221377 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.221381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.222835 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.222846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.223724 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.223728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.223834 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.223872 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:11:18.223995 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 03:11:18.224121 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":68,"thread_start_us":118,"threads_started":1}
I20260929 03:11:18.224558 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.217693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:32907 (local address 127.0.0.1:53094)
0929 03:11:18.218127 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:18.218195 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:18.218325 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:18.219581 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:18.219587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:18.219612 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:11:18.219818 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.219826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.221400 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.221403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.222948 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.222959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.224098 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.224102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.224216 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.224242 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:11:18.224316 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:11:18.224396 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":87,"thread_start_us":115,"threads_started":1}
I20260929 03:11:18.225359 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.217957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.20.174.67:32907)
0929 03:11:18.219039 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:18.219054 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:18.219080 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:11:18.219166 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:18.219171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:18.219391 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:18.219564 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:11:18.219970 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.221211 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.223098 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.223957 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.224780 (+ 823us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.224979 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:18.225041 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:11:18.225142 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":133,"threads_started":1}
I20260929 03:11:18.225502 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.218086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:44969 (local address 127.0.0.1:55532)
0929 03:11:18.218665 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:18.218679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:18.218766 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:18.220523 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:18.220526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:18.220545 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:11:18.220641 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.220648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.222093 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.222097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.223315 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.223325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.225072 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.225076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.225164 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.225185 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:11:18.225278 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:11:18.225361 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":19,"thread_start_us":109,"threads_started":1}
I20260929 03:11:18.226572 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.219419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.20.174.66:44969)
0929 03:11:18.220123 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:18.220148 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:18.220165 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:11:18.220232 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:18.220236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:18.220376 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:18.220538 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:11:18.220764 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.221930 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.223456 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.224880 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.225073 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.225207 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:18.225308 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 03:11:18.226389 (+ 1081us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":108,"threads_started":1}
I20260929 03:11:18.227869 31558 tablet_service.cc:1467] Processing CreateTablet for tablet 503bd0053fdf4bb88e702fde385759db (DEFAULT_TABLE table=test-table [id=4579ebb365324b78a928689f02a5cccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:11:18.228245 31826 tablet_service.cc:1467] Processing CreateTablet for tablet 503bd0053fdf4bb88e702fde385759db (DEFAULT_TABLE table=test-table [id=4579ebb365324b78a928689f02a5cccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:11:18.229116 31558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 503bd0053fdf4bb88e702fde385759db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:11:18.229185 31826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 503bd0053fdf4bb88e702fde385759db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:11:18.229353 31692 tablet_service.cc:1467] Processing CreateTablet for tablet 503bd0053fdf4bb88e702fde385759db (DEFAULT_TABLE table=test-table [id=4579ebb365324b78a928689f02a5cccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:11:18.230378 31692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 503bd0053fdf4bb88e702fde385759db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:11:18.236083 31909 tablet_bootstrap.cc:492] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: Bootstrap starting.
I20260929 03:11:18.237422 31911 tablet_bootstrap.cc:492] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: Bootstrap starting.
I20260929 03:11:18.238806 31910 tablet_bootstrap.cc:492] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e: Bootstrap starting.
I20260929 03:11:18.238894 31909 tablet_bootstrap.cc:654] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: Neither blocks nor log segments found. Creating new log.
I20260929 03:11:18.240170 31911 tablet_bootstrap.cc:654] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: Neither blocks nor log segments found. Creating new log.
I20260929 03:11:18.240320 31909 log.cc:824] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: Log is configured to *not* fsync() on all Append() calls
I20260929 03:11:18.241010 31911 log.cc:824] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: Log is configured to *not* fsync() on all Append() calls
I20260929 03:11:18.241089 31910 tablet_bootstrap.cc:654] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e: Neither blocks nor log segments found. Creating new log.
I20260929 03:11:18.241961 31910 log.cc:824] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:11:18.242769 31909 tablet_bootstrap.cc:492] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: No bootstrap required, opened a new log
I20260929 03:11:18.242978 31909 ts_tablet_manager.cc:1397] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 03:11:18.243314 31911 tablet_bootstrap.cc:492] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: No bootstrap required, opened a new log
I20260929 03:11:18.243497 31911 ts_tablet_manager.cc:1397] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:11:18.243868 31910 tablet_bootstrap.cc:492] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e: No bootstrap required, opened a new log
I20260929 03:11:18.244040 31910 ts_tablet_manager.cc:1397] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:11:18.247157 31909 raft_consensus.cc:348] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cc85e37da544a895532038d1b1af65" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32907 } } peers { permanent_uuid: "737042732c374706bdd28e170967122e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 32897 } } peers { permanent_uuid: "4b9186f09cd0413c8aca153014a035ed" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44969 } }
I20260929 03:11:18.247481 31909 raft_consensus.cc:374] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:11:18.247545 31909 raft_consensus.cc:729] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66cc85e37da544a895532038d1b1af65, State: Initialized, Role: FOLLOWER
I20260929 03:11:18.247550 31911 raft_consensus.cc:348] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cc85e37da544a895532038d1b1af65" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32907 } } peers { permanent_uuid: "737042732c374706bdd28e170967122e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 32897 } } peers { permanent_uuid: "4b9186f09cd0413c8aca153014a035ed" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44969 } }
I20260929 03:11:18.247558 31910 raft_consensus.cc:348] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cc85e37da544a895532038d1b1af65" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32907 } } peers { permanent_uuid: "737042732c374706bdd28e170967122e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 32897 } } peers { permanent_uuid: "4b9186f09cd0413c8aca153014a035ed" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44969 } }
I20260929 03:11:18.247828 31911 raft_consensus.cc:374] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:11:18.247860 31910 raft_consensus.cc:374] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:11:18.247893 31911 raft_consensus.cc:729] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9186f09cd0413c8aca153014a035ed, State: Initialized, Role: FOLLOWER
I20260929 03:11:18.247934 31910 raft_consensus.cc:729] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737042732c374706bdd28e170967122e, State: Initialized, Role: FOLLOWER
I20260929 03:11:18.248283 31909 consensus_queue.cc:260] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cc85e37da544a895532038d1b1af65" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32907 } } peers { permanent_uuid: "737042732c374706bdd28e170967122e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 32897 } } peers { permanent_uuid: "4b9186f09cd0413c8aca153014a035ed" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44969 } }
I20260929 03:11:18.248283 31911 consensus_queue.cc:260] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cc85e37da544a895532038d1b1af65" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32907 } } peers { permanent_uuid: "737042732c374706bdd28e170967122e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 32897 } } peers { permanent_uuid: "4b9186f09cd0413c8aca153014a035ed" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44969 } }
I20260929 03:11:18.248406 31910 consensus_queue.cc:260] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cc85e37da544a895532038d1b1af65" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32907 } } peers { permanent_uuid: "737042732c374706bdd28e170967122e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 32897 } } peers { permanent_uuid: "4b9186f09cd0413c8aca153014a035ed" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44969 } }
I20260929 03:11:18.249035 31758 heartbeater.cc:503] Master 127.20.174.126:33741 was elected leader, sending a full tablet report...
I20260929 03:11:18.249392 31909 ts_tablet_manager.cc:1428] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260929 03:11:18.249404 31910 ts_tablet_manager.cc:1428] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:11:18.249732 31892 heartbeater.cc:503] Master 127.20.174.126:33741 was elected leader, sending a full tablet report...
I20260929 03:11:18.249737 31624 heartbeater.cc:503] Master 127.20.174.126:33741 was elected leader, sending a full tablet report...
I20260929 03:11:18.251731 31911 ts_tablet_manager.cc:1428] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260929 03:11:18.414302 31893 tablet.cc:2369] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:11:18.443753 31759 tablet.cc:2369] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:11:18.461282 31625 tablet.cc:2369] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:11:18.572392 31916 raft_consensus.cc:482] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:11:18.572669 31916 raft_consensus.cc:504] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cc85e37da544a895532038d1b1af65" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32907 } } peers { permanent_uuid: "737042732c374706bdd28e170967122e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 32897 } } peers { permanent_uuid: "4b9186f09cd0413c8aca153014a035ed" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44969 } }
I20260929 03:11:18.573926 31916 leader_election.cc:288] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66cc85e37da544a895532038d1b1af65 (127.20.174.67:32907), 737042732c374706bdd28e170967122e (127.20.174.65:32897)
I20260929 03:11:18.579649 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.574795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:32897 (local address 127.20.174.66:35885)
0929 03:11:18.575122 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:18.575135 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:18.575265 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:18.575510 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:18.575512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:18.575526 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:11:18.575653 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.575659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.576628 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.576630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.578067 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.578078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.579117 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.579120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.579279 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.579294 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 03:11:18.579360 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:11:18.579459 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":77,"thread_start_us":89,"threads_started":1}
I20260929 03:11:18.579653 31918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.574373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:32907 (local address 127.20.174.66:46845)
0929 03:11:18.574707 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:18.574722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:11:18.574986 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:11:18.575359 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:11:18.575362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:11:18.575382 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:11:18.575527 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.575534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.576603 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.576618 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.578067 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:11:18.578078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.579117 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.579120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:11:18.579254 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.579279 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:11:18.579360 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:11:18.579459 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":118,"thread_start_us":144,"threads_started":1}
I20260929 03:11:18.579936 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.574972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:35885 (local address 127.20.174.65:32897)
0929 03:11:18.575135 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:18.575140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:18.575231 (+ 91us) server_negotiation.cc:407] Connection header received
0929 03:11:18.575359 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:18.575362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:18.575406 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:18.575488 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:11:18.575768 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.576504 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.578215 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.578868 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.578966 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.579017 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:18.579408 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 03:11:18.579567 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:11:18.580538 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.574504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:46845 (local address 127.20.174.67:32907)
0929 03:11:18.574677 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:18.574683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:18.574895 (+ 212us) server_negotiation.cc:407] Connection header received
0929 03:11:18.575143 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:18.575149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:18.575246 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:18.575330 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:11:18.575645 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.576474 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.578215 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.578868 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.578966 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.579017 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:18.580331 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0929 03:11:18.580416 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:11:18.580578 31578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "503bd0053fdf4bb88e702fde385759db" candidate_uuid: "4b9186f09cd0413c8aca153014a035ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737042732c374706bdd28e170967122e" is_pre_election: true
I20260929 03:11:18.580917 31578 raft_consensus.cc:2454] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b9186f09cd0413c8aca153014a035ed in term 0.
I20260929 03:11:18.581187 31846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "503bd0053fdf4bb88e702fde385759db" candidate_uuid: "4b9186f09cd0413c8aca153014a035ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66cc85e37da544a895532038d1b1af65" is_pre_election: true
I20260929 03:11:18.581435 31647 leader_election.cc:302] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b9186f09cd0413c8aca153014a035ed, 737042732c374706bdd28e170967122e; no voters:
I20260929 03:11:18.581487 31846 raft_consensus.cc:2454] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b9186f09cd0413c8aca153014a035ed in term 0.
I20260929 03:11:18.581708 31916 raft_consensus.cc:2781] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:11:18.581837 31916 raft_consensus.cc:482] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:11:18.581892 31916 raft_consensus.cc:3037] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:11:18.583098 31916 raft_consensus.cc:504] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cc85e37da544a895532038d1b1af65" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32907 } } peers { permanent_uuid: "737042732c374706bdd28e170967122e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 32897 } } peers { permanent_uuid: "4b9186f09cd0413c8aca153014a035ed" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44969 } }
I20260929 03:11:18.583454 31916 leader_election.cc:288] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [CANDIDATE]: Term 1 election: Requested vote from peers 66cc85e37da544a895532038d1b1af65 (127.20.174.67:32907), 737042732c374706bdd28e170967122e (127.20.174.65:32897)
I20260929 03:11:18.583798 31846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "503bd0053fdf4bb88e702fde385759db" candidate_uuid: "4b9186f09cd0413c8aca153014a035ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66cc85e37da544a895532038d1b1af65"
I20260929 03:11:18.583827 31578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "503bd0053fdf4bb88e702fde385759db" candidate_uuid: "4b9186f09cd0413c8aca153014a035ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737042732c374706bdd28e170967122e"
I20260929 03:11:18.583910 31846 raft_consensus.cc:3037] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:11:18.583981 31578 raft_consensus.cc:3037] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:11:18.585186 31578 raft_consensus.cc:2454] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9186f09cd0413c8aca153014a035ed in term 1.
I20260929 03:11:18.585204 31846 raft_consensus.cc:2454] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9186f09cd0413c8aca153014a035ed in term 1.
I20260929 03:11:18.585495 31647 leader_election.cc:302] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b9186f09cd0413c8aca153014a035ed, 737042732c374706bdd28e170967122e; no voters:
I20260929 03:11:18.585697 31916 raft_consensus.cc:2781] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:11:18.585987 31916 raft_consensus.cc:686] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 1 LEADER]: Becoming Leader. State: Replica: 4b9186f09cd0413c8aca153014a035ed, State: Running, Role: LEADER
I20260929 03:11:18.586345 31916 consensus_queue.cc:237] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cc85e37da544a895532038d1b1af65" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32907 } } peers { permanent_uuid: "737042732c374706bdd28e170967122e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 32897 } } peers { permanent_uuid: "4b9186f09cd0413c8aca153014a035ed" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44969 } }
I20260929 03:11:18.589797 31433 catalog_manager.cc:5495] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b9186f09cd0413c8aca153014a035ed (127.20.174.66). New cstate: current_term: 1 leader_uuid: "4b9186f09cd0413c8aca153014a035ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cc85e37da544a895532038d1b1af65" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737042732c374706bdd28e170967122e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 32897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b9186f09cd0413c8aca153014a035ed" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44969 } health_report { overall_health: HEALTHY } } }
I20260929 03:11:18.688655 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.684486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.20.174.66:44969)
0929 03:11:18.684655 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:11:18.684661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:11:18.684677 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:11:18.684769 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:11:18.684771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:11:18.684837 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:11:18.684965 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:11:18.685321 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.685961 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.686989 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:11:18.687235 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:11:18.687367 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:11:18.687725 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:11:18.688199 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:11:18.688357 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 03:11:18.688426 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:11:18.688489 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:11:18.695520 31846 raft_consensus.cc:1261] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65 [term 1 FOLLOWER]: Refusing update from remote peer 4b9186f09cd0413c8aca153014a035ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:11:18.695520 31578 raft_consensus.cc:1261] T 503bd0053fdf4bb88e702fde385759db P 737042732c374706bdd28e170967122e [term 1 FOLLOWER]: Refusing update from remote peer 4b9186f09cd0413c8aca153014a035ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:11:18.696419 31916 consensus_queue.cc:1035] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "737042732c374706bdd28e170967122e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:11:18.696868 31920 consensus_queue.cc:1035] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "66cc85e37da544a895532038d1b1af65" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:11:18.708541 31925 mvcc.cc:205] Tried to move back new op lower bound from 7334508456727695360 to 7334508456292478976. Current Snapshot: MvccSnapshot[applied={T|T < 7334508456727695360}]
I20260929 03:11:18.714346 31432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:11:18.714583 31432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:11:18.717531 31432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d69824dc03040b48b3fc016c0f27306: Sending DeleteTablet for 3 replicas of tablet 503bd0053fdf4bb88e702fde385759db
I20260929 03:11:18.718568 31826 tablet_service.cc:1514] Processing DeleteTablet for tablet 503bd0053fdf4bb88e702fde385759db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:11:18 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:53094
I20260929 03:11:18.718600 31692 tablet_service.cc:1514] Processing DeleteTablet for tablet 503bd0053fdf4bb88e702fde385759db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:11:18 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:55532
I20260929 03:11:18.718717 31558 tablet_service.cc:1514] Processing DeleteTablet for tablet 503bd0053fdf4bb88e702fde385759db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:11:18 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:37978
I20260929 03:11:18.719828 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 31495
I20260929 03:11:18.720641 31933 tablet_replica.cc:330] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: stopping tablet replica
I20260929 03:11:18.721037 31933 raft_consensus.cc:2229] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:11:18.721475 31933 raft_consensus.cc:2258] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:11:18.721908 31934 tablet_replica.cc:330] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: stopping tablet replica
I20260929 03:11:18.722216 31934 raft_consensus.cc:2229] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:11:18.722638 31934 raft_consensus.cc:2258] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:11:18.724344 31934 ts_tablet_manager.cc:1905] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:11:18.726692 31933 ts_tablet_manager.cc:1905] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:11:18.728117 31934 ts_tablet_manager.cc:1918] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:11:18.728266 31934 log.cc:1196] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/503bd0053fdf4bb88e702fde385759db
I20260929 03:11:18.728602 31934 ts_tablet_manager.cc:1939] T 503bd0053fdf4bb88e702fde385759db P 4b9186f09cd0413c8aca153014a035ed: Deleting consensus metadata
I20260929 03:11:18.729537 31933 ts_tablet_manager.cc:1918] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:11:18.729643 31933 log.cc:1196] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/503bd0053fdf4bb88e702fde385759db
I20260929 03:11:18.729907 31418 catalog_manager.cc:4841] TS 4b9186f09cd0413c8aca153014a035ed (127.20.174.66:44969): tablet 503bd0053fdf4bb88e702fde385759db (table test-table [id=4579ebb365324b78a928689f02a5cccd]) successfully deleted
I20260929 03:11:18.729965 31933 ts_tablet_manager.cc:1939] T 503bd0053fdf4bb88e702fde385759db P 66cc85e37da544a895532038d1b1af65: Deleting consensus metadata
I20260929 03:11:18.731253 31419 catalog_manager.cc:4841] TS 66cc85e37da544a895532038d1b1af65 (127.20.174.67:32907): tablet 503bd0053fdf4bb88e702fde385759db (table test-table [id=4579ebb365324b78a928689f02a5cccd]) successfully deleted
W20260929 03:11:18.733687 31419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:32897 (error 108)
I20260929 03:11:18.733778 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 31629
I20260929 03:11:18.735910 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:11:18.735316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:32897 (local address 127.0.0.1:37986)
0929 03:11:18.735656 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 03:11:18.735752 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:32897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":69,"threads_started":1}
W20260929 03:11:18.736101 31419 catalog_manager.cc:4584] TS 737042732c374706bdd28e170967122e (127.20.174.65:32897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 503bd0053fdf4bb88e702fde385759db: Network error: Client connection negotiation failed: client connection to 127.20.174.65:32897: connect: Connection refused (error 111)
I20260929 03:11:18.739650 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 31763
I20260929 03:11:18.745297 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 31401
2026-09-29T03:11:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2711 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:11:18 dist-test-slave-k1dg krb5kdc[31940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:11:18 dist-test-slave-k1dg krb5kdc[31940](info): set up 2 sockets
Sep 29 03:11:18 dist-test-slave-k1dg krb5kdc[31940](info): commencing operation
krb5kdc: starting...
W20260929 03:11:19.787489 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:11:19 dist-test-slave-k1dg krb5kdc[31940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:11:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:11:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/rangeradmin_127.20.174.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/rangeradmin_127.20.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/rangerlookup_127.20.174.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/rangerlookup_127.20.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/HTTP_127.20.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/HTTP_127.20.174.84@KRBTEST.COM.keytab.
I20260929 03:11:19.969247 21177 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskEa40KE/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/postgres -l logfile start
2026-09-29 03:11:22.740 UTC [31970] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:11:22.740 UTC [31970] LOG: listening on IPv4 address "127.20.174.84", port 59793
2026-09-29 03:11:22.745 UTC [31970] LOG: listening on Unix socket "/tmp/.s.PGSQL.59793"
2026-09-29 03:11:22.782 UTC [31973] LOG: database system was shut down at 2026-09-29 03:11:21 UTC
2026-09-29 03:11:22.786 UTC [31970] LOG: database system is ready to accept connections
I20260929 03:11:23.767390 21177 mini_postgres.cc:96] Postgres bound to 59793
2026-09-29 03:11:23.774 UTC [31981] FATAL: database "slave" does not exist
127.20.174.84:59793 - accepting connections
I20260929 03:11:23.774933 21177 mini_ranger.cc:161] Starting Ranger...
I20260929 03:11:23.790933 21177 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:11:24.372551 21177 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:11:24.372653 21177 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-admin
2026-09-29 03:11:24,661 [I] DB FLAVOR :POSTGRES
2026-09-29 03:11:24,662 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:11:24,662 [I] Checking connection..
2026-09-29 03:11:24,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:11:24,920 [I] Checking connection passed.
2026-09-29 03:11:24,921 [I] --------- Verifying version history table ---------
2026-09-29 03:11:24,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:11:25,157 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:11:25,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:11:25,431 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:11:25,432 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:11:25,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:11:25.695 UTC [32045] WARNING: there is no transaction in progress
2026-09-29 03:11:25,702 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:11:25,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:11:25,962 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:11:25,962 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:11:25,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:11:26,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:11:26,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k1dg.c.gcp-upstream.internal','N') ;"
2026-09-29 03:11:26,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:11:26,998 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:11:26,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:11:27,273 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:11:27,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:11:27,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:11:27,811 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:11:27,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:11:28.985 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.007 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.030 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.052 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.073 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.094 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.162 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.171 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.191 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.204 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.219 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.239 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.253 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.264 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.275 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:29.286 UTC [32153] WARNING: there is no transaction in progress
2026-09-29 03:11:30,152 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:11:30,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:11:30,390 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:11:30,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:11:30,641 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:11:30,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:11:30,924 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:11:30,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:11:31,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:11:31,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k1dg.c.gcp-upstream.internal';"
2026-09-29 03:11:31,724 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:11:31,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:11:31,967 [I] DB_PATCHES have already been applied
I20260929 03:11:31.973352 21177 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-admin:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskEa40KE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:11:31.973488 21177 mini_ranger.cc:193] Using host: 127.20.174.84
I20260929 03:11:31.977273 21177 mini_ranger.cc:227] Ranger admin URL: http://127.20.174.84:57957
Sep 29, 2026 3:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:11:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:11:33.106105 21177 mini_ranger.cc:160] Time spent starting Ranger: real 9.331s user 0.000s sys 0.011s
Sep 29, 2026 3:11:54 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:11:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 3:11:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:11:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:11:55 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:11:55.779933 21177 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/rangerkms_127.20.174.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/rangerkms_127.20.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.174.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.174.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.174.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/HTTP_127.20.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.174.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/HTTP_127.20.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:11:55 dist-test-slave-k1dg krb5kdc[31940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651515, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:11:55.886616 21177 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:11:55.898914 21177 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:11:56.978166 21177 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:11:56.978273 21177 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-kms
I20260929 03:11:56.978379 21177 mini_ranger_kms.cc:227] Using postgres at 127.20.174.84:59793
2026-09-29 03:11:57,168 [I] DB FLAVOR :POSTGRES
2026-09-29 03:11:57,168 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:11:57,168 [I] Checking connection
2026-09-29 03:11:57,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:11:57,445 [I] connection success
2026-09-29 03:11:57,445 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:11:57,445 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:11:57,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:11:57,715 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:11:57,715 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:11:57,715 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:11:57,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:59793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:11:58,016 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:11:58.589383 21177 mini_ranger_kms.cc:310] Created kms service
I20260929 03:11:58.834002 21177 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:11:58.885828 21177 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:11:58 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651515, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:11:59.321702 21177 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:11:59.321861 21177 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-kms:/tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:11:59.321882 21177 mini_ranger_kms.cc:240] Using host: 127.20.174.84
I20260929 03:11:59.325891 21177 mini_ranger_kms.cc:276] Ranger KMS PID: 32334
I20260929 03:11:59.326054 21177 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.174.84:43141
Sep 29, 2026 3:12:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:12:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:12:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:12:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:12:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:12:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:12:01.673733 21177 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.787s user 0.008s sys 0.165s
I20260929 03:12:01.673880 21177 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:12:01.674026 21177 mini_ranger_kms.cc:401] 127.20.174.84:43141/kms/v1/keys
Sep 29 03:12:01 dist-test-slave-k1dg krb5kdc[31940](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651521, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.174.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:12:02 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651521, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.174.84@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
Sep 29 03:12:02 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651521, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.174.84@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
Sep 29 03:12:05 dist-test-slave-k1dg krb5kdc[31940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:12:05.066821 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:34013
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:34013
--ranger_config_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:43141/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:12:05.186893 32384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:05.187201 32384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:05.187240 32384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:05.191516 32384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:12:05.191601 32384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:05.191622 32384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:05.191638 32384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:12:05.191708 32384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:12:05.197650 32384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.174.84:43141/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:34013
--ranger_config_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:34013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:05.198947 32384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:05.200060 32384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:12:05.210240 32384 server_base.cc:1034] running on GCE node
W20260929 03:12:05.210273 32392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:05.210150 32389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:05.210155 32390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:05.211161 32384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:05.212443 32384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:05.213634 32384 hybrid_clock.cc:648] HybridClock initialized: now 1790651525213598 us; error 50 us; skew 500 ppm
Sep 29 03:12:05 dist-test-slave-k1dg krb5kdc[31940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651525, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:05.224490 32384 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:12:05.225621 32384 webserver.cc:460] Webserver started at http://127.20.174.126:46769/ using document root <none> and password file <none>
I20260929 03:12:05.226301 32384 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:05.226390 32384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:12:05.226651 32384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:12:05 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651525, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:05.300895 32384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "08ce7b91858a4d23aa3a10dd95cfc3de"
format_stamp: "Formatted at 2026-09-29 03:12:05 on dist-test-slave-k1dg"
server_key: "c368dc8d5fb7846fbabf4e1613315615"
server_key_iv: "7bfc7af98843df0968195d480969f133"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:05.301563 32384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "08ce7b91858a4d23aa3a10dd95cfc3de"
format_stamp: "Formatted at 2026-09-29 03:12:05 on dist-test-slave-k1dg"
server_key: "c368dc8d5fb7846fbabf4e1613315615"
server_key_iv: "7bfc7af98843df0968195d480969f133"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:05.305800 32384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:12:05.325688 32399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:05.327239 32384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:12:05.327402 32384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "08ce7b91858a4d23aa3a10dd95cfc3de"
format_stamp: "Formatted at 2026-09-29 03:12:05 on dist-test-slave-k1dg"
server_key: "c368dc8d5fb7846fbabf4e1613315615"
server_key_iv: "7bfc7af98843df0968195d480969f133"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:05.327546 32384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:05.349730 32384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:05.352962 32384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:05.353178 32384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:05.362373 32384 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:34013
I20260929 03:12:05.363052 32451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:34013 every 8 connection(s)
I20260929 03:12:05.363727 32384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:12:05.366477 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 32384
I20260929 03:12:05.366617 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:12:05.366983 21177 external_mini_cluster.cc:1428] Setting key e942f6a7759dae459095643c391b7c3f
I20260929 03:12:05.367111 32452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:12:05.374782 32452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de: Bootstrap starting.
Sep 29 03:12:05 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:12:05.377681 32452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de: Neither blocks nor log segments found. Creating new log.
I20260929 03:12:05.378765 32452 log.cc:824] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de: Log is configured to *not* fsync() on all Append() calls
I20260929 03:12:05.381347 32452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de: No bootstrap required, opened a new log
I20260929 03:12:05.384575 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:05.368487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.20.174.126:34013)
0929 03:12:05.368930 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:05.368952 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:05.368998 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:12:05.370104 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:05.370127 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:05.370576 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:05.370893 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:05.371945 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:05.373224 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:05.374564 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:05.375175 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:05.375309 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:05.378529 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:05.378536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:05.378553 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:05.378586 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:05.381303 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:05.381927 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:05.381932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:05.381937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:05.382054 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:05.382393 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:05.382395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:05.382397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:05.382921 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:05.383079 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:05.383416 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 03:12:05.383705 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":159,"threads_started":1}
I20260929 03:12:05.385257 32452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ce7b91858a4d23aa3a10dd95cfc3de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34013 } }
I20260929 03:12:05.385491 32452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:12:05.385552 32452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ce7b91858a4d23aa3a10dd95cfc3de, State: Initialized, Role: FOLLOWER
I20260929 03:12:05.386024 32452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ce7b91858a4d23aa3a10dd95cfc3de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34013 } }
I20260929 03:12:05.386170 32452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:12:05.386256 32452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:12:05.386381 32452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:12:05.387543 32452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ce7b91858a4d23aa3a10dd95cfc3de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34013 } }
I20260929 03:12:05.387926 32452 leader_election.cc:302] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08ce7b91858a4d23aa3a10dd95cfc3de; no voters:
I20260929 03:12:05.388270 32452 leader_election.cc:288] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:12:05.388412 32457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:12:05.388676 32457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [term 1 LEADER]: Becoming Leader. State: Replica: 08ce7b91858a4d23aa3a10dd95cfc3de, State: Running, Role: LEADER
I20260929 03:12:05.389243 32457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ce7b91858a4d23aa3a10dd95cfc3de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34013 } }
I20260929 03:12:05.389629 32452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:12:05.390524 32457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ce7b91858a4d23aa3a10dd95cfc3de. Latest consensus state: current_term: 1 leader_uuid: "08ce7b91858a4d23aa3a10dd95cfc3de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ce7b91858a4d23aa3a10dd95cfc3de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34013 } } }
I20260929 03:12:05.390758 32457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [sys.catalog]: This master's current role is: LEADER
I20260929 03:12:05.391006 32458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08ce7b91858a4d23aa3a10dd95cfc3de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ce7b91858a4d23aa3a10dd95cfc3de" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 34013 } } }
I20260929 03:12:05.391104 32458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de [sys.catalog]: This master's current role is: LEADER
I20260929 03:12:05.391324 32466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:12:05.394660 32466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:12:05.395331 32452 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:12:05.405000 32466 catalog_manager.cc:1347] Generated new cluster ID: 88bd913e05354f6ab252da3640857732
I20260929 03:12:05.405089 32466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:12:05.418018 32466 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:12:05.419559 32466 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:12:05.430598 32466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de: Generated new TSK 0
I20260929 03:12:05.431630 32466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:12:06.171041 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:43141/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34013
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:12:06.302758 32497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:06.303081 32497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:06.303195 32497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:06.307581 32497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:06.307677 32497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:06.307823 32497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:12:06.313699 32497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.174.84:43141/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:06.315102 32497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:06.316246 32497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:12:06.327387 32503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:06.327391 32505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:06.327845 32502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:06.328516 32497 server_base.cc:1034] running on GCE node
I20260929 03:12:06.329124 32497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:06.330067 32497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:06.331286 32497 hybrid_clock.cc:648] HybridClock initialized: now 1790651526331246 us; error 60 us; skew 500 ppm
Sep 29 03:12:06 dist-test-slave-k1dg krb5kdc[31940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651526, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:06.335395 32497 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:12:06.336586 32497 webserver.cc:460] Webserver started at http://127.20.174.65:46365/ using document root <none> and password file <none>
I20260929 03:12:06.337234 32497 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:06.337342 32497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:12:06.337625 32497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:12:06 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651526, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:06.359701 32497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "ead59d22e9eb4f79b49a844f6028b619"
format_stamp: "Formatted at 2026-09-29 03:12:06 on dist-test-slave-k1dg"
server_key: "bf9bbf956f35ec7f321034abd4f6f7a1"
server_key_iv: "6d69e8eb70775bbfeec5058acfd8eb83"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:06.360601 32497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "ead59d22e9eb4f79b49a844f6028b619"
format_stamp: "Formatted at 2026-09-29 03:12:06 on dist-test-slave-k1dg"
server_key: "bf9bbf956f35ec7f321034abd4f6f7a1"
server_key_iv: "6d69e8eb70775bbfeec5058acfd8eb83"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:06.369267 32497 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260929 03:12:06.388480 32514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:06.392211 32497 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260929 03:12:06.392532 32497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "ead59d22e9eb4f79b49a844f6028b619"
format_stamp: "Formatted at 2026-09-29 03:12:06 on dist-test-slave-k1dg"
server_key: "bf9bbf956f35ec7f321034abd4f6f7a1"
server_key_iv: "6d69e8eb70775bbfeec5058acfd8eb83"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:06.392836 32497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:06.410410 32497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:06.417802 32497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:06.418030 32497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:06.418857 32497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:12:06.420010 32497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:12:06.420120 32497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:06.420238 32497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:12:06.420282 32497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:06.433661 32497 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:33653
I20260929 03:12:06.433718 32627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:33653 every 8 connection(s)
I20260929 03:12:06.435230 32497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:12:06.437139 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 32497
I20260929 03:12:06.437304 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:12:06.437697 21177 external_mini_cluster.cc:1428] Setting key 95b195bf451fc655183a1e81fedcdd8b
Sep 29 03:12:06 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651526, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:12:06.454470 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:06.437543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:39525 (local address 127.20.174.126:34013)
0929 03:12:06.437825 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:06.437830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:06.438827 (+ 997us) server_negotiation.cc:407] Connection header received
0929 03:12:06.439859 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:06.439862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:06.439916 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:06.440037 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:06.441238 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:06.442122 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:06.444689 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:06.445109 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:06.445213 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:06.448989 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:06.448997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:06.449000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:06.449041 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:06.451249 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:06.452163 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:06.452168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:06.452170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:06.452302 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:06.452828 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:06.452832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:06.452834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:06.453106 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:06.453227 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:06.454040 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 03:12:06.454227 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
I20260929 03:12:06.456324 32631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:06.438047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:34013 (local address 127.20.174.65:39525)
0929 03:12:06.438665 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:06.438726 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:06.439626 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:06.440240 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:06.440248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:06.440536 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:12:06.441073 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:06.441087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:06.442313 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:06.442318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:06.444492 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:06.444510 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:06.445476 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:06.445481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:06.445731 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:06.446609 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:12:06.446635 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:12:06.448705 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:12:06.451440 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:06.451447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:06.451468 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:06.452007 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:06.452423 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:06.452426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:06.452429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:06.452660 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:06.453294 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:12:06.453302 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:12:06.453712 (+ 410us) client_negotiation.cc:769] Sending connection context
0929 03:12:06.454674 (+ 962us) client_negotiation.cc:240] Negotiation successful
0929 03:12:06.455088 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":135,"threads_started":1}
I20260929 03:12:06.458055 32628 heartbeater.cc:347] Connected to a master server at 127.20.174.126:34013
I20260929 03:12:06.458374 32628 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:06.459127 32628 heartbeater.cc:511] Master 127.20.174.126:34013 requested a full tablet report, sending...
I20260929 03:12:06.461169 32416 ts_manager.cc:194] Registered new tserver with Master: ead59d22e9eb4f79b49a844f6028b619 (127.20.174.65:33653)
I20260929 03:12:06.462980 32416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:39525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:12:06.485627 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:43141/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34013
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:12:06 dist-test-slave-k1dg krb5kdc[31940](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651526, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 03:12:06.667421 32634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:06.667908 32634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:06.667990 32634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:06.674772 32634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:06.674870 32634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:06.674966 32634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:12:06.683943 32634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.174.84:43141/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:06.685401 32634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:06.686925 32634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:12:06.699780 32645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:06.699920 32642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:06.699805 32643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:06.700655 32634 server_base.cc:1034] running on GCE node
I20260929 03:12:06.701152 32634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:06.701853 32634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:06.703049 32634 hybrid_clock.cc:648] HybridClock initialized: now 1790651526703009 us; error 62 us; skew 500 ppm
Sep 29 03:12:06 dist-test-slave-k1dg krb5kdc[31940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651526, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:06.706254 32634 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:12:06.707417 32634 webserver.cc:460] Webserver started at http://127.20.174.66:45463/ using document root <none> and password file <none>
I20260929 03:12:06.708122 32634 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:06.708225 32634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:12:06.708487 32634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:12:06 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651526, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:06.725025 32634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "90ec102dd9da4c55945d43d645b7d7b0"
format_stamp: "Formatted at 2026-09-29 03:12:06 on dist-test-slave-k1dg"
server_key: "c746f5ffe85a817179a8e33c519e821a"
server_key_iv: "244f100e837e28e7faf008f76e6ce49f"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:06.725878 32634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "90ec102dd9da4c55945d43d645b7d7b0"
format_stamp: "Formatted at 2026-09-29 03:12:06 on dist-test-slave-k1dg"
server_key: "c746f5ffe85a817179a8e33c519e821a"
server_key_iv: "244f100e837e28e7faf008f76e6ce49f"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:06.731567 32634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 03:12:06.754912 32652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:06.756976 32634 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260929 03:12:06.757157 32634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "90ec102dd9da4c55945d43d645b7d7b0"
format_stamp: "Formatted at 2026-09-29 03:12:06 on dist-test-slave-k1dg"
server_key: "c746f5ffe85a817179a8e33c519e821a"
server_key_iv: "244f100e837e28e7faf008f76e6ce49f"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:06.757339 32634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:06.784487 32634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:06.789014 32634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:06.789319 32634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:06.790309 32634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:12:06.791600 32634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:12:06.791733 32634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:06.792002 32634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:12:06.792079 32634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:06.806828 32634 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:44867
I20260929 03:12:06.808187 32634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:12:06.808444 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 32634
I20260929 03:12:06.808573 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:12:06.808939 21177 external_mini_cluster.cc:1428] Setting key ed6cdfd5c270ab5b5382c9167bb4a830
I20260929 03:12:06.809901 32765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:44867 every 8 connection(s)
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
Sep 29 03:12:06 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651526, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:12:06.841524 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:06.813899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:45937 (local address 127.20.174.126:34013)
0929 03:12:06.814034 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:06.814039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:06.819092 (+ 5053us) server_negotiation.cc:407] Connection header received
0929 03:12:06.820922 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:06.820927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:06.821004 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:06.821075 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:06.827430 (+ 6355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:06.828338 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:06.833014 (+ 4676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:06.833392 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:06.833485 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:06.836949 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:06.836954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:06.836957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:06.836989 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:06.838844 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:06.839562 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:06.839567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:06.839569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:06.839668 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:06.840127 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:06.840130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:06.840132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:06.840368 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:06.840456 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:06.841190 (+ 734us) server_negotiation.cc:299] Negotiation successful
0929 03:12:06.841342 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:12:06.843623 301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:06.812705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:34013 (local address 127.20.174.66:45937)
0929 03:12:06.818848 (+ 6143us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:06.818943 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:06.820474 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:06.821626 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:06.821639 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:06.822130 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:12:06.823026 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:06.823051 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:06.828725 (+ 5674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:06.828729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:06.832807 (+ 4078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:06.832831 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:06.833742 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:06.833746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:06.834010 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:06.834810 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:12:06.834833 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:12:06.836685 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:12:06.839018 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:06.839025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:06.839046 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:06.839431 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:06.839800 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:06.839803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:06.839805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:06.840012 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:06.840527 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:12:06.840534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:12:06.840899 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 03:12:06.842034 (+ 1135us) client_negotiation.cc:240] Negotiation successful
0929 03:12:06.842463 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5835,"thread_start_us":151,"threads_started":1}
I20260929 03:12:06.848217 32766 heartbeater.cc:347] Connected to a master server at 127.20.174.126:34013
I20260929 03:12:06.848490 32766 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:06.849123 32766 heartbeater.cc:511] Master 127.20.174.126:34013 requested a full tablet report, sending...
I20260929 03:12:06.850463 32416 ts_manager.cc:194] Registered new tserver with Master: 90ec102dd9da4c55945d43d645b7d7b0 (127.20.174.66:44867)
I20260929 03:12:06.851263 32416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:45937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:12:06.871660 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:43141/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34013
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:12:07.022827 303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:07.023378 303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:07.023514 303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:07.029120 303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:07.029290 303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:07.029435 303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:12:07.036492 303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.174.84:43141/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:07.038121 303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:07.039691 303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:12:07.052866 309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:07.052877 308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:07.053422 311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:07.054322 303 server_base.cc:1034] running on GCE node
I20260929 03:12:07.054920 303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:07.055824 303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:07.057046 303 hybrid_clock.cc:648] HybridClock initialized: now 1790651527057032 us; error 41 us; skew 500 ppm
Sep 29 03:12:07 dist-test-slave-k1dg krb5kdc[31940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651527, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:07.060999 303 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:12:07.062299 303 webserver.cc:460] Webserver started at http://127.20.174.67:44049/ using document root <none> and password file <none>
I20260929 03:12:07.063118 303 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:07.063242 303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:12:07.063532 303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:12:07 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651527, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:07.086768 303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "09e371d712364d73be12551880a67dbe"
format_stamp: "Formatted at 2026-09-29 03:12:07 on dist-test-slave-k1dg"
server_key: "c9ef35edc33825e29037a11cc58db041"
server_key_iv: "7d2c28007c2cb7cf24f4be0ddef0aae4"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:07.087667 303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "09e371d712364d73be12551880a67dbe"
format_stamp: "Formatted at 2026-09-29 03:12:07 on dist-test-slave-k1dg"
server_key: "c9ef35edc33825e29037a11cc58db041"
server_key_iv: "7d2c28007c2cb7cf24f4be0ddef0aae4"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:07.095494 303 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260929 03:12:07.125316 318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Sep 29 03:12:07 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651526, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:07.127525 303 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.000s
I20260929 03:12:07.127720 303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "09e371d712364d73be12551880a67dbe"
format_stamp: "Formatted at 2026-09-29 03:12:07 on dist-test-slave-k1dg"
server_key: "c9ef35edc33825e29037a11cc58db041"
server_key_iv: "7d2c28007c2cb7cf24f4be0ddef0aae4"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:07.127890 303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:07.162462 303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:07.166972 303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:07.167250 303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:07.168123 303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:12:07.170013 303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:12:07.170120 303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:07.170177 303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:12:07.170274 303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 03:12:07.185202 303 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:39905
I20260929 03:12:07.185457 431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:39905 every 8 connection(s)
I20260929 03:12:07.186724 303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:12:07.196458 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 303
I20260929 03:12:07.196615 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:12:07.196959 21177 external_mini_cluster.cc:1428] Setting key e3c51fc7e9120fc8ba1d8b36efa79a6b
Sep 29 03:12:07 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651527, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:12:07.205811 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.189052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:60833 (local address 127.20.174.126:34013)
0929 03:12:07.189195 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:07.189199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:07.190525 (+ 1326us) server_negotiation.cc:407] Connection header received
0929 03:12:07.192281 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:07.192285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:07.192350 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:07.192453 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:07.194170 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.195044 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.197097 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.197444 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.197535 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.200959 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:07.200963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:07.200966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:07.200999 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:07.202944 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:07.203712 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:07.203716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:07.203718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:07.203821 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:07.204311 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:07.204314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:07.204316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:07.204566 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:07.204649 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:07.205471 (+ 822us) server_negotiation.cc:299] Negotiation successful
0929 03:12:07.205634 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:12:07.208187 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.189650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:34013 (local address 127.20.174.67:60833)
0929 03:12:07.190271 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:07.190348 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:07.191989 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:07.192795 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:07.192805 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:07.193220 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:12:07.193963 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.193989 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.195235 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.195239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.196858 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.196884 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.197765 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.197769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.198002 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.198827 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:12:07.198850 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:12:07.200715 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:12:07.203122 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:07.203129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:07.203149 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:07.203573 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:07.203968 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:07.203970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:07.203972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:07.204185 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:07.204742 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:12:07.204749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:12:07.205142 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 03:12:07.206372 (+ 1230us) client_negotiation.cc:240] Negotiation successful
0929 03:12:07.206845 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":169,"threads_started":1}
I20260929 03:12:07.209975 432 heartbeater.cc:347] Connected to a master server at 127.20.174.126:34013
I20260929 03:12:07.210254 432 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:07.210943 432 heartbeater.cc:511] Master 127.20.174.126:34013 requested a full tablet report, sending...
I20260929 03:12:07.212218 32416 ts_manager.cc:194] Registered new tserver with Master: 09e371d712364d73be12551880a67dbe (127.20.174.67:39905)
I20260929 03:12:07.212983 32416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:60833
I20260929 03:12:07.224277 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:12:07.237620 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.226795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.20.174.126:34013)
0929 03:12:07.226936 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:07.226941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:07.227663 (+ 722us) server_negotiation.cc:407] Connection header received
0929 03:12:07.227909 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:07.227913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:07.227966 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:07.228022 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:07.229069 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.229829 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.231121 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.231468 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.231542 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.233760 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:07.233765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:07.233768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:07.233802 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:07.235691 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:07.236241 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:07.236245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:07.236247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:07.236335 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:07.236624 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:07.236628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:07.236629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:07.236842 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:07.236960 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:07.237320 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 03:12:07.237450 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:12:07.242027 32416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:12:07.247033 32416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:12:07.275936 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.258233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:39905 (local address 127.0.0.1:51104)
0929 03:12:07.266688 (+ 8455us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:07.266765 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:07.266929 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:07.269617 (+ 2688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:07.269625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:07.269648 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:12:07.269792 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.269798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.271283 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.271287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.272799 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.272813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.275306 (+ 2493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.275310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.275446 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.275492 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:12:07.275623 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 03:12:07.275763 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8184,"thread_start_us":101,"threads_started":1}
I20260929 03:12:07.276901 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.259136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.20.174.67:39905)
0929 03:12:07.267847 (+ 8711us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:07.267862 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:07.267892 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:12:07.267983 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:07.267988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:07.269234 (+ 1246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:07.269424 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:12:07.269931 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.271129 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.274007 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.274956 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.275068 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.275238 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:07.276619 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0929 03:12:07.276744 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8580,"thread_start_us":118,"threads_started":1}
I20260929 03:12:07.276906 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.259625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:33653 (local address 127.0.0.1:52292)
0929 03:12:07.267175 (+ 7550us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:07.267188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:07.267263 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:07.271328 (+ 4065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:07.271332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:07.271353 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:12:07.271459 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.271465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.273297 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.273301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.274573 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.274585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.276507 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.276511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.276610 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.276635 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:12:07.276682 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 03:12:07.276754 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7487,"thread_start_us":86,"threads_started":1}
I20260929 03:12:07.280756 352 tablet_service.cc:1467] Processing CreateTablet for tablet acfa129ccded458c9d4c0c7a1636a3f9 (DEFAULT_TABLE table=test-table [id=1f5b4d14352e46e6b611b0d837991eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:12:07.282194 352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfa129ccded458c9d4c0c7a1636a3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:12:07.294412 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.260012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.20.174.66:44867)
0929 03:12:07.275929 (+ 15917us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:07.275946 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:07.275973 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:12:07.276055 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:07.276061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:07.276314 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:07.277512 (+ 1198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:12:07.289395 (+ 11883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.290787 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.292511 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.293431 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.293523 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.293690 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:07.294080 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 03:12:07.294239 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15767,"thread_start_us":117,"threads_started":1}
I20260929 03:12:07.295131 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.259840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:44867 (local address 127.0.0.1:57276)
0929 03:12:07.267441 (+ 7601us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:07.267453 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:07.267521 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:07.288990 (+ 21469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:07.288994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:07.289027 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:12:07.289175 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.289196 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.290980 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.290983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.292310 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.292324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.293804 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.293807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.293916 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.293940 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:12:07.294895 (+ 955us) client_negotiation.cc:240] Negotiation successful
0929 03:12:07.294994 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7548,"thread_start_us":111,"threads_started":1}
I20260929 03:12:07.296811 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.260021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.20.174.65:33653)
0929 03:12:07.270657 (+ 10636us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:07.270665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:07.270694 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:12:07.270774 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:07.270779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:07.271114 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:07.271631 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:12:07.271772 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.273113 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.274746 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.275607 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.277220 (+ 1613us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.296484 (+ 19264us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:07.296552 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 03:12:07.296668 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10511,"thread_start_us":95,"threads_started":1}
I20260929 03:12:07.299075 32700 tablet_service.cc:1467] Processing CreateTablet for tablet acfa129ccded458c9d4c0c7a1636a3f9 (DEFAULT_TABLE table=test-table [id=1f5b4d14352e46e6b611b0d837991eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:12:07.300577 32700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfa129ccded458c9d4c0c7a1636a3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:12:07.300311 32562 tablet_service.cc:1467] Processing CreateTablet for tablet acfa129ccded458c9d4c0c7a1636a3f9 (DEFAULT_TABLE table=test-table [id=1f5b4d14352e46e6b611b0d837991eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:12:07.301379 32562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acfa129ccded458c9d4c0c7a1636a3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:12:07.310695 451 tablet_bootstrap.cc:492] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619: Bootstrap starting.
I20260929 03:12:07.313648 451 tablet_bootstrap.cc:654] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619: Neither blocks nor log segments found. Creating new log.
I20260929 03:12:07.314826 451 log.cc:824] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619: Log is configured to *not* fsync() on all Append() calls
I20260929 03:12:07.315838 450 tablet_bootstrap.cc:492] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0: Bootstrap starting.
I20260929 03:12:07.319818 450 tablet_bootstrap.cc:654] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0: Neither blocks nor log segments found. Creating new log.
I20260929 03:12:07.320967 450 log.cc:824] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:12:07.321949 453 tablet_bootstrap.cc:492] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe: Bootstrap starting.
I20260929 03:12:07.327317 451 tablet_bootstrap.cc:492] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619: No bootstrap required, opened a new log
I20260929 03:12:07.327673 451 ts_tablet_manager.cc:1397] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260929 03:12:07.335585 451 raft_consensus.cc:348] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e371d712364d73be12551880a67dbe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39905 } } peers { permanent_uuid: "ead59d22e9eb4f79b49a844f6028b619" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33653 } } peers { permanent_uuid: "90ec102dd9da4c55945d43d645b7d7b0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44867 } }
I20260929 03:12:07.336216 451 raft_consensus.cc:374] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:12:07.336963 453 tablet_bootstrap.cc:654] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe: Neither blocks nor log segments found. Creating new log.
I20260929 03:12:07.337139 451 raft_consensus.cc:729] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead59d22e9eb4f79b49a844f6028b619, State: Initialized, Role: FOLLOWER
I20260929 03:12:07.337934 451 consensus_queue.cc:260] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e371d712364d73be12551880a67dbe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39905 } } peers { permanent_uuid: "ead59d22e9eb4f79b49a844f6028b619" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33653 } } peers { permanent_uuid: "90ec102dd9da4c55945d43d645b7d7b0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44867 } }
I20260929 03:12:07.339560 453 log.cc:824] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe: Log is configured to *not* fsync() on all Append() calls
I20260929 03:12:07.341342 451 ts_tablet_manager.cc:1428] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260929 03:12:07.343258 450 tablet_bootstrap.cc:492] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0: No bootstrap required, opened a new log
I20260929 03:12:07.343581 450 ts_tablet_manager.cc:1397] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20260929 03:12:07.348345 450 raft_consensus.cc:348] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e371d712364d73be12551880a67dbe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39905 } } peers { permanent_uuid: "ead59d22e9eb4f79b49a844f6028b619" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33653 } } peers { permanent_uuid: "90ec102dd9da4c55945d43d645b7d7b0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44867 } }
I20260929 03:12:07.349797 32628 heartbeater.cc:503] Master 127.20.174.126:34013 was elected leader, sending a full tablet report...
I20260929 03:12:07.355567 450 raft_consensus.cc:374] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:12:07.355784 450 raft_consensus.cc:729] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ec102dd9da4c55945d43d645b7d7b0, State: Initialized, Role: FOLLOWER
I20260929 03:12:07.356494 450 consensus_queue.cc:260] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e371d712364d73be12551880a67dbe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39905 } } peers { permanent_uuid: "ead59d22e9eb4f79b49a844f6028b619" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33653 } } peers { permanent_uuid: "90ec102dd9da4c55945d43d645b7d7b0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44867 } }
I20260929 03:12:07.357908 450 ts_tablet_manager.cc:1428] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260929 03:12:07.358368 32766 heartbeater.cc:503] Master 127.20.174.126:34013 was elected leader, sending a full tablet report...
I20260929 03:12:07.365896 453 tablet_bootstrap.cc:492] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe: No bootstrap required, opened a new log
I20260929 03:12:07.366253 453 ts_tablet_manager.cc:1397] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.000s
I20260929 03:12:07.370554 453 raft_consensus.cc:348] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e371d712364d73be12551880a67dbe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39905 } } peers { permanent_uuid: "ead59d22e9eb4f79b49a844f6028b619" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33653 } } peers { permanent_uuid: "90ec102dd9da4c55945d43d645b7d7b0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44867 } }
I20260929 03:12:07.371199 453 raft_consensus.cc:374] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:12:07.373206 453 raft_consensus.cc:729] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e371d712364d73be12551880a67dbe, State: Initialized, Role: FOLLOWER
I20260929 03:12:07.373935 453 consensus_queue.cc:260] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e371d712364d73be12551880a67dbe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39905 } } peers { permanent_uuid: "ead59d22e9eb4f79b49a844f6028b619" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33653 } } peers { permanent_uuid: "90ec102dd9da4c55945d43d645b7d7b0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44867 } }
I20260929 03:12:07.375324 453 ts_tablet_manager.cc:1428] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260929 03:12:07.375811 432 heartbeater.cc:503] Master 127.20.174.126:34013 was elected leader, sending a full tablet report...
I20260929 03:12:07.418882 455 raft_consensus.cc:482] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:12:07.419139 455 raft_consensus.cc:504] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e371d712364d73be12551880a67dbe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39905 } } peers { permanent_uuid: "ead59d22e9eb4f79b49a844f6028b619" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33653 } } peers { permanent_uuid: "90ec102dd9da4c55945d43d645b7d7b0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44867 } }
I20260929 03:12:07.420729 455 leader_election.cc:288] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09e371d712364d73be12551880a67dbe (127.20.174.67:39905), 90ec102dd9da4c55945d43d645b7d7b0 (127.20.174.66:44867)
I20260929 03:12:07.428321 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.421866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:35327 (local address 127.20.174.66:44867)
0929 03:12:07.422006 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:07.422014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:07.422979 (+ 965us) server_negotiation.cc:407] Connection header received
0929 03:12:07.423226 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:07.423230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:07.423289 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:07.423348 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:12:07.423849 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.424697 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.426740 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.427448 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.427575 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.427639 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:07.428057 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 03:12:07.428150 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:12:07.429097 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.421383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:44867 (local address 127.20.174.65:35327)
0929 03:12:07.422872 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:07.422889 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:07.423074 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:07.423522 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:07.423525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:07.423547 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:12:07.423698 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.423706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.424910 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.424914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.426546 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.426562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.427805 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.427809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.427909 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.427932 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:12:07.428823 (+ 891us) client_negotiation.cc:240] Negotiation successful
0929 03:12:07.428925 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"thread_start_us":141,"threads_started":1}
I20260929 03:12:07.430117 32705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfa129ccded458c9d4c0c7a1636a3f9" candidate_uuid: "ead59d22e9eb4f79b49a844f6028b619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90ec102dd9da4c55945d43d645b7d7b0" is_pre_election: true
I20260929 03:12:07.430544 32705 raft_consensus.cc:2454] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ead59d22e9eb4f79b49a844f6028b619 in term 0.
I20260929 03:12:07.431357 32517 leader_election.cc:302] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90ec102dd9da4c55945d43d645b7d7b0, ead59d22e9eb4f79b49a844f6028b619; no voters:
I20260929 03:12:07.431789 455 raft_consensus.cc:2781] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:12:07.431911 455 raft_consensus.cc:482] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:12:07.431972 455 raft_consensus.cc:3037] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:12:07.433600 455 raft_consensus.cc:504] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e371d712364d73be12551880a67dbe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39905 } } peers { permanent_uuid: "ead59d22e9eb4f79b49a844f6028b619" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33653 } } peers { permanent_uuid: "90ec102dd9da4c55945d43d645b7d7b0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44867 } }
I20260929 03:12:07.434116 455 leader_election.cc:288] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [CANDIDATE]: Term 1 election: Requested vote from peers 09e371d712364d73be12551880a67dbe (127.20.174.67:39905), 90ec102dd9da4c55945d43d645b7d7b0 (127.20.174.66:44867)
I20260929 03:12:07.435097 32705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfa129ccded458c9d4c0c7a1636a3f9" candidate_uuid: "ead59d22e9eb4f79b49a844f6028b619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90ec102dd9da4c55945d43d645b7d7b0"
I20260929 03:12:07.435261 32705 raft_consensus.cc:3037] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:12:07.436748 32705 raft_consensus.cc:2454] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ead59d22e9eb4f79b49a844f6028b619 in term 1.
W20260929 03:12:07.436977 32629 tablet.cc:2369] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:12:07.437021 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.422634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:36373 (local address 127.20.174.67:39905)
0929 03:12:07.422790 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:07.422795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:07.431600 (+ 8805us) server_negotiation.cc:407] Connection header received
0929 03:12:07.432103 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:07.432107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:07.432172 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:07.432234 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:12:07.432798 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.433659 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.435471 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.436139 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.436256 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.436315 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:07.436781 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 03:12:07.436876 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:12:07.437276 32517 leader_election.cc:302] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90ec102dd9da4c55945d43d645b7d7b0, ead59d22e9eb4f79b49a844f6028b619; no voters:
I20260929 03:12:07.437498 460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.422155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:39905 (local address 127.20.174.65:36373)
0929 03:12:07.431486 (+ 9331us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:07.431505 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:07.431914 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:07.432483 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:07.432486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:07.432506 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:12:07.432638 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.432644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.433913 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.433916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.435292 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:07.435305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.436432 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.436436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:07.436569 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.436614 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 03:12:07.437234 (+ 620us) client_negotiation.cc:240] Negotiation successful
0929 03:12:07.437342 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9238,"thread_start_us":104,"threads_started":1}
I20260929 03:12:07.437762 455 raft_consensus.cc:2781] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:12:07.438033 455 raft_consensus.cc:686] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [term 1 LEADER]: Becoming Leader. State: Replica: ead59d22e9eb4f79b49a844f6028b619, State: Running, Role: LEADER
I20260929 03:12:07.438282 368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfa129ccded458c9d4c0c7a1636a3f9" candidate_uuid: "ead59d22e9eb4f79b49a844f6028b619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e371d712364d73be12551880a67dbe" is_pre_election: true
I20260929 03:12:07.438478 455 consensus_queue.cc:237] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e371d712364d73be12551880a67dbe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39905 } } peers { permanent_uuid: "ead59d22e9eb4f79b49a844f6028b619" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33653 } } peers { permanent_uuid: "90ec102dd9da4c55945d43d645b7d7b0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44867 } }
I20260929 03:12:07.438634 368 raft_consensus.cc:2454] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ead59d22e9eb4f79b49a844f6028b619 in term 0.
I20260929 03:12:07.439525 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfa129ccded458c9d4c0c7a1636a3f9" candidate_uuid: "ead59d22e9eb4f79b49a844f6028b619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e371d712364d73be12551880a67dbe"
I20260929 03:12:07.439662 370 raft_consensus.cc:3037] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:12:07.441200 370 raft_consensus.cc:2454] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ead59d22e9eb4f79b49a844f6028b619 in term 1.
W20260929 03:12:07.441720 433 tablet.cc:2369] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:12:07.445156 32416 catalog_manager.cc:5495] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 reported cstate change: term changed from 0 to 1, leader changed from <none> to ead59d22e9eb4f79b49a844f6028b619 (127.20.174.65). New cstate: current_term: 1 leader_uuid: "ead59d22e9eb4f79b49a844f6028b619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09e371d712364d73be12551880a67dbe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ead59d22e9eb4f79b49a844f6028b619" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90ec102dd9da4c55945d43d645b7d7b0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44867 } health_report { overall_health: UNKNOWN } } }
Sep 29 03:12:07 dist-test-slave-k1dg krb5kdc[31940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651526, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:07.508543 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.502630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.20.174.65:33653)
0929 03:12:07.502794 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:07.502800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:07.503201 (+ 401us) server_negotiation.cc:407] Connection header received
0929 03:12:07.503299 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:07.503302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:07.503369 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:07.503473 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:12:07.503877 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.504548 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.505973 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:07.506357 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:07.506515 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:07.507037 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:12:07.507728 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:12:07.507871 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:07.508299 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 03:12:07.508378 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:12:07.516788 32705 raft_consensus.cc:1261] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0 [term 1 FOLLOWER]: Refusing update from remote peer ead59d22e9eb4f79b49a844f6028b619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:12:07.517993 461 consensus_queue.cc:1035] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90ec102dd9da4c55945d43d645b7d7b0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:12:07.518491 370 raft_consensus.cc:1261] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe [term 1 FOLLOWER]: Refusing update from remote peer ead59d22e9eb4f79b49a844f6028b619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:12:07.522833 461 consensus_queue.cc:1035] T acfa129ccded458c9d4c0c7a1636a3f9 P ead59d22e9eb4f79b49a844f6028b619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e371d712364d73be12551880a67dbe" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:12:07.536998 468 mvcc.cc:205] Tried to move back new op lower bound from 7334508656699514880 to 7334508656402542592. Current Snapshot: MvccSnapshot[applied={T|T < 7334508656699514880}]
I20260929 03:12:07.541556 465 mvcc.cc:205] Tried to move back new op lower bound from 7334508656699514880 to 7334508656402542592. Current Snapshot: MvccSnapshot[applied={T|T < 7334508656699514880 or (T in {7334508656699514880})}]
I20260929 03:12:07.542809 466 mvcc.cc:205] Tried to move back new op lower bound from 7334508656699514880 to 7334508656402542592. Current Snapshot: MvccSnapshot[applied={T|T < 7334508656699514880}]
I20260929 03:12:07.547458 32416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:12:07.547660 32416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:12:07.551606 32416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08ce7b91858a4d23aa3a10dd95cfc3de: Sending DeleteTablet for 3 replicas of tablet acfa129ccded458c9d4c0c7a1636a3f9
I20260929 03:12:07.552821 32562 tablet_service.cc:1514] Processing DeleteTablet for tablet acfa129ccded458c9d4c0c7a1636a3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:12:07 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:52292
I20260929 03:12:07.553828 32700 tablet_service.cc:1514] Processing DeleteTablet for tablet acfa129ccded458c9d4c0c7a1636a3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:12:07 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:57276
I20260929 03:12:07.554569 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 32497
I20260929 03:12:07.554600 352 tablet_service.cc:1514] Processing DeleteTablet for tablet acfa129ccded458c9d4c0c7a1636a3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:12:07 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:51104
I20260929 03:12:07.555766 476 tablet_replica.cc:330] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe: stopping tablet replica
I20260929 03:12:07.556186 476 raft_consensus.cc:2229] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:12:07.556519 476 raft_consensus.cc:2258] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:12:07.560974 475 tablet_replica.cc:330] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0: stopping tablet replica
I20260929 03:12:07.562237 475 raft_consensus.cc:2229] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:12:07.562707 475 raft_consensus.cc:2258] T acfa129ccded458c9d4c0c7a1636a3f9 P 90ec102dd9da4c55945d43d645b7d7b0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 03:12:07.565825 32400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:33653 (error 108)
I20260929 03:12:07.566820 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 32634
I20260929 03:12:07.569044 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.568181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:33653 (local address 127.0.0.1:52306)
0929 03:12:07.568826 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:07.568922 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525}
W20260929 03:12:07.569257 32400 catalog_manager.cc:4584] TS ead59d22e9eb4f79b49a844f6028b619 (127.20.174.65:33653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet acfa129ccded458c9d4c0c7a1636a3f9: Network error: Client connection negotiation failed: client connection to 127.20.174.65:33653: connect: Connection refused (error 111)
I20260929 03:12:07.577992 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:07.577621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:44867 (local address 127.0.0.1:57286)
0929 03:12:07.577825 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:07.577886 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.66:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 03:12:07.578061 476 ts_tablet_manager.cc:1905] T acfa129ccded458c9d4c0c7a1636a3f9 P 09e371d712364d73be12551880a67dbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:12:07.581823 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 303
W20260929 03:12:07.591172 447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:12:07.590056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:39905 (local address 127.0.0.1:51112)
0929 03:12:07.590264 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:07.590280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:07.590375 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:07.591035 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.67:39905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 03:12:07.591884 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 32384
I20260929 03:12:07.600201 21177 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:12:07.946394 21177 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:12:07.946607 21177 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:12:08.292742 21177 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:12:08.293 UTC [31970] LOG: received smart shutdown request
2026-09-29 03:12:08.297 UTC [31970] LOG: background worker "logical replication launcher" (PID 31979) exited with exit code 1
2026-09-29 03:12:08.298 UTC [31974] LOG: shutting down
2026-09-29 03:12:08.517 UTC [31970] LOG: database system is shut down
2026-09-29T03:12:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49896 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:12:08 dist-test-slave-k1dg krb5kdc[487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:12:08 dist-test-slave-k1dg krb5kdc[487](info): set up 2 sockets
Sep 29 03:12:08 dist-test-slave-k1dg krb5kdc[487](info): commencing operation
krb5kdc: starting...
W20260929 03:12:09.746767 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:12:09 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:12:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/rangeradmin_127.20.174.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/rangeradmin_127.20.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/rangerlookup_127.20.174.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/rangerlookup_127.20.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/HTTP_127.20.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/HTTP_127.20.174.84@KRBTEST.COM.keytab.
I20260929 03:12:09.937536 21177 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskEa40KE/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/postgres -l logfile start
2026-09-29 03:12:12.927 UTC [517] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:12:12.927 UTC [517] LOG: listening on IPv4 address "127.20.174.84", port 37959
2026-09-29 03:12:12.931 UTC [517] LOG: listening on Unix socket "/tmp/.s.PGSQL.37959"
2026-09-29 03:12:12.955 UTC [520] LOG: database system was shut down at 2026-09-29 03:12:11 UTC
2026-09-29 03:12:12.961 UTC [517] LOG: database system is ready to accept connections
I20260929 03:12:13.938661 21177 mini_postgres.cc:96] Postgres bound to 37959
2026-09-29 03:12:13.945 UTC [528] FATAL: database "slave" does not exist
127.20.174.84:37959 - accepting connections
I20260929 03:12:13.946120 21177 mini_ranger.cc:161] Starting Ranger...
I20260929 03:12:13.962711 21177 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:12:14.584028 21177 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:12:14.584138 21177 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-admin
2026-09-29 03:12:14,862 [I] DB FLAVOR :POSTGRES
2026-09-29 03:12:14,862 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:12:14,863 [I] Checking connection..
2026-09-29 03:12:14,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:12:15,117 [I] Checking connection passed.
2026-09-29 03:12:15,118 [I] --------- Verifying version history table ---------
2026-09-29 03:12:15,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:12:15,378 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:12:15,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:12:15,629 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:12:15,629 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:12:15,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:12:15.914 UTC [592] WARNING: there is no transaction in progress
2026-09-29 03:12:15,921 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:12:15,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:12:16,194 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:12:16,195 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:12:16,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:12:16,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:12:16,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k1dg.c.gcp-upstream.internal','N') ;"
2026-09-29 03:12:16,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:12:17,221 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:12:17,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:12:17,481 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:12:17,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:12:17,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:12:17,995 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:12:17,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:12:19.230 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.250 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.273 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.295 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.315 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.336 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.410 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.419 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.437 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.449 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.461 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.479 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.491 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.501 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.511 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:19.520 UTC [700] WARNING: there is no transaction in progress
2026-09-29 03:12:20,396 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:12:20,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:12:20,634 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:12:20,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:12:20,893 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:12:20,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:12:21,172 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:12:21,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:12:21,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:12:21,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k1dg.c.gcp-upstream.internal';"
2026-09-29 03:12:21,918 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:12:21,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:12:22,208 [I] DB_PATCHES have already been applied
I20260929 03:12:22.214850 21177 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-admin:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskEa40KE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:12:22.214922 21177 mini_ranger.cc:193] Using host: 127.20.174.84
I20260929 03:12:22.218667 21177 mini_ranger.cc:227] Ranger admin URL: http://127.20.174.84:59385
Sep 29, 2026 3:12:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:12:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:12:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:12:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:12:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:12:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:12:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:12:23.340015 21177 mini_ranger.cc:160] Time spent starting Ranger: real 9.394s user 0.005s sys 0.005s
Sep 29, 2026 3:12:45 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:12:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 3:12:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:12:45 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:12:46 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:12:46.901108 21177 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.174.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.174.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/rangerkms_127.20.174.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.174.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/rangerkms_127.20.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.174.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.174.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.174.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/HTTP_127.20.174.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.174.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/HTTP_127.20.174.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:12:47 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651567, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:12:47.008824 21177 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:12:47.019758 21177 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:12:48.026270 21177 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:12:48.026388 21177 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-kms
I20260929 03:12:48.026412 21177 mini_ranger_kms.cc:227] Using postgres at 127.20.174.84:37959
2026-09-29 03:12:48,217 [I] DB FLAVOR :POSTGRES
2026-09-29 03:12:48,217 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:12:48,217 [I] Checking connection
2026-09-29 03:12:48,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:12:48,466 [I] connection success
2026-09-29 03:12:48,466 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:12:48,466 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:12:48,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:12:48,725 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:12:48,725 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:12:48,725 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:12:48,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.174.84:37959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:12:49,030 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:12:49.508661 21177 mini_ranger_kms.cc:310] Created kms service
I20260929 03:12:49.680979 21177 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:12:49.728159 21177 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:12:49 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651567, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:50.138751 21177 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:12:50.138919 21177 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-kms:/tmp/dist-test-taskEa40KE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskEa40KE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:12:50.138943 21177 mini_ranger_kms.cc:240] Using host: 127.20.174.84
I20260929 03:12:50.142931 21177 mini_ranger_kms.cc:276] Ranger KMS PID: 881
I20260929 03:12:50.143033 21177 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.174.84:39679
Sep 29, 2026 3:12:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:12:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:12:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:12:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:12:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:12:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:12:52.422422 21177 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.414s user 0.014s sys 0.160s
I20260929 03:12:52.422572 21177 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:12:52.422638 21177 mini_ranger_kms.cc:401] 127.20.174.84:39679/kms/v1/keys
Sep 29 03:12:52 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651572, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.174.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:12:53 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651572, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.174.84@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
Sep 29 03:12:53 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651572, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.174.84@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
Sep 29 03:12:55 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:12:55.853673 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:39777
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:39777
--ranger_config_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:39679/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:12:55.993673 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:55.994138 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:55.994288 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:56.001741 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:12:56.001844 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:56.001883 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:56.001941 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:12:56.001995 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:12:56.012946 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.174.84:39679/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:39777
--ranger_config_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:39777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:56.014803 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:56.016536 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:12:56.028950 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:56.029058 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:56.029129 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:56.036258 931 server_base.cc:1034] running on GCE node
I20260929 03:12:56.037364 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:56.039186 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:56.040431 931 hybrid_clock.cc:648] HybridClock initialized: now 1790651576040403 us; error 43 us; skew 500 ppm
Sep 29 03:12:56 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651576, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:56.044559 931 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:12:56.045639 931 webserver.cc:460] Webserver started at http://127.20.174.126:41103/ using document root <none> and password file <none>
I20260929 03:12:56.046272 931 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:56.046332 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:12:56.046605 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:12:56 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651576, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:56.103375 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "a1cb3d743c38483ca25a22b30c5ea349"
format_stamp: "Formatted at 2026-09-29 03:12:56 on dist-test-slave-k1dg"
server_key: "86435eff21ad2fbfc9486d41a7a6cdd5"
server_key_iv: "2cfbbac915ecf24e3c1c248c641cee38"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:56.104075 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "a1cb3d743c38483ca25a22b30c5ea349"
format_stamp: "Formatted at 2026-09-29 03:12:56 on dist-test-slave-k1dg"
server_key: "86435eff21ad2fbfc9486d41a7a6cdd5"
server_key_iv: "2cfbbac915ecf24e3c1c248c641cee38"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:56.108425 931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:12:56.127027 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:56.128854 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:12:56.129011 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "a1cb3d743c38483ca25a22b30c5ea349"
format_stamp: "Formatted at 2026-09-29 03:12:56 on dist-test-slave-k1dg"
server_key: "86435eff21ad2fbfc9486d41a7a6cdd5"
server_key_iv: "2cfbbac915ecf24e3c1c248c641cee38"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:56.129158 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:56.156075 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:56.159552 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:56.159751 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:56.168162 931 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:39777
I20260929 03:12:56.168694 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:39777 every 8 connection(s)
I20260929 03:12:56.169405 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:12:56.173075 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:12:56.173734 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 931
I20260929 03:12:56.173992 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:12:56.174376 21177 external_mini_cluster.cc:1428] Setting key ac6974d50b870595e362476b8d8ce7ff
I20260929 03:12:56.180392 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349: Bootstrap starting.
I20260929 03:12:56.183187 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349: Neither blocks nor log segments found. Creating new log.
Sep 29 03:12:56 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:12:56.184095 999 log.cc:824] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349: Log is configured to *not* fsync() on all Append() calls
I20260929 03:12:56.186508 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349: No bootstrap required, opened a new log
I20260929 03:12:56.190693 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } }
I20260929 03:12:56.190975 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:12:56.191054 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1cb3d743c38483ca25a22b30c5ea349, State: Initialized, Role: FOLLOWER
I20260929 03:12:56.191121 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:56.175984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.20.174.126:39777)
0929 03:12:56.176457 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:56.176474 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:56.176512 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:12:56.177216 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:56.177234 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:56.177606 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:56.177983 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:56.178927 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:56.180104 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:56.181157 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:56.181678 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:56.181882 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:56.184864 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:56.184870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:56.184889 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:56.184919 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:56.187682 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:56.188265 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:56.188272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:56.188279 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:56.188427 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:56.188760 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:56.188763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:56.188764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:56.189273 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:56.189477 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:56.189813 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 03:12:56.190106 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":166,"threads_started":1}
I20260929 03:12:56.191629 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } }
I20260929 03:12:56.191805 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:12:56.191891 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:12:56.192034 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:12:56.193363 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } }
I20260929 03:12:56.193750 999 leader_election.cc:302] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1cb3d743c38483ca25a22b30c5ea349; no voters:
I20260929 03:12:56.194111 999 leader_election.cc:288] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:12:56.194247 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:12:56.194504 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 1 LEADER]: Becoming Leader. State: Replica: a1cb3d743c38483ca25a22b30c5ea349, State: Running, Role: LEADER
I20260929 03:12:56.194880 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } }
I20260929 03:12:56.195568 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:12:56.196182 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1cb3d743c38483ca25a22b30c5ea349. Latest consensus state: current_term: 1 leader_uuid: "a1cb3d743c38483ca25a22b30c5ea349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } } }
I20260929 03:12:56.196944 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [sys.catalog]: This master's current role is: LEADER
I20260929 03:12:56.196833 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1cb3d743c38483ca25a22b30c5ea349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } } }
I20260929 03:12:56.197233 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [sys.catalog]: This master's current role is: LEADER
I20260929 03:12:56.197531 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:12:56.198082 999 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:12:56.204771 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:12:56.216482 1013 catalog_manager.cc:1347] Generated new cluster ID: 62ab49d81d2c47478159b5473f674586
I20260929 03:12:56.216563 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:12:56.241958 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:12:56.243681 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:12:56.260655 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349: Generated new TSK 0
I20260929 03:12:56.261798 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:12:57.014679 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:39679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39777
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:12:57.140333 1044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:57.140658 1044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:57.140799 1044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:57.145215 1044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:57.145318 1044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:57.145426 1044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:12:57.151194 1044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.174.84:39679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:57.152436 1044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:57.153523 1044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:12:57.164026 1049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:57.164458 1050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:57.164498 1044 server_base.cc:1034] running on GCE node
W20260929 03:12:57.165222 1052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:57.165851 1044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:57.166781 1044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:57.172622 1044 hybrid_clock.cc:648] HybridClock initialized: now 1790651577172570 us; error 62 us; skew 500 ppm
Sep 29 03:12:57 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651577, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:57.177165 1044 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:12:57.178369 1044 webserver.cc:460] Webserver started at http://127.20.174.65:33741/ using document root <none> and password file <none>
I20260929 03:12:57.179147 1044 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:57.179243 1044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:12:57.179540 1044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:12:57 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651577, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:57.198689 1044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "0c35421495e049d1b0b4487f06cc63ab"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "69ba1a6e50f82dfe246877fc205effa4"
server_key_iv: "3dedb4133d200253671f64d8044f8cde"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:57.199767 1044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "0c35421495e049d1b0b4487f06cc63ab"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "69ba1a6e50f82dfe246877fc205effa4"
server_key_iv: "3dedb4133d200253671f64d8044f8cde"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:57.206096 1044 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 03:12:57.224596 1060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:57.231935 1044 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260929 03:12:57.232211 1044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "0c35421495e049d1b0b4487f06cc63ab"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "69ba1a6e50f82dfe246877fc205effa4"
server_key_iv: "3dedb4133d200253671f64d8044f8cde"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:57.232476 1044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:57.265085 1044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:57.269477 1044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:57.269754 1044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:57.270797 1044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:12:57.272154 1044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:12:57.272282 1044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:57.272418 1044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:12:57.272508 1044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:57.284556 1044 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:43601
I20260929 03:12:57.284795 1174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:43601 every 8 connection(s)
I20260929 03:12:57.286021 1044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:12:57.291968 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 1044
I20260929 03:12:57.292096 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:12:57.292397 21177 external_mini_cluster.cc:1428] Setting key 439030447ad207d40e425dd60a74d58e
Sep 29 03:12:57 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651577, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:12:57.317230 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:57.292241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:45639 (local address 127.20.174.126:39777)
0929 03:12:57.292978 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:57.292983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:57.293001 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:12:57.293047 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:57.293050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:57.293101 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:57.293208 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:57.297055 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:57.297968 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:57.299891 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:57.300255 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:57.300364 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:57.312239 (+ 11875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:57.312244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:57.312247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:57.312288 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:57.314405 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:57.315224 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:57.315228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:57.315230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:57.315327 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:57.315772 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:57.315776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:57.315777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:57.316029 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:57.316124 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:57.316871 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 03:12:57.317034 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":107,"threads_started":1}
I20260929 03:12:57.319428 1177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:57.288995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39777 (local address 127.20.174.65:45639)
0929 03:12:57.290413 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:57.290508 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:57.291687 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:57.293560 (+ 1873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:57.293572 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:57.293972 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:12:57.296816 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:57.296841 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:57.298158 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:57.298163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:57.299719 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:57.299734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:57.300575 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:57.300580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:57.300802 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:57.309846 (+ 9044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:12:57.309875 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:12:57.311859 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:12:57.314623 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:57.314632 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:57.314654 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:57.315089 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:57.315434 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:57.315437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:57.315439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:57.315657 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:57.316184 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:12:57.316191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:12:57.316565 (+ 374us) client_negotiation.cc:769] Sending connection context
0929 03:12:57.317778 (+ 1213us) client_negotiation.cc:240] Negotiation successful
0929 03:12:57.318195 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":168,"threads_started":1}
I20260929 03:12:57.321539 1175 heartbeater.cc:347] Connected to a master server at 127.20.174.126:39777
I20260929 03:12:57.321810 1175 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:57.322472 1175 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
I20260929 03:12:57.324653 956 ts_manager.cc:194] Registered new tserver with Master: 0c35421495e049d1b0b4487f06cc63ab (127.20.174.65:43601)
I20260929 03:12:57.326407 956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:45639
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:12:57.366856 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:39679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39777
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:12:57 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651577, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 03:12:57.549000 1183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:57.549510 1183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:57.549626 1183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:57.556412 1183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:57.556525 1183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:57.556627 1183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:12:57.565326 1183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.174.84:39679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:57.567013 1183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:57.568496 1183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:12:57.582527 1190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:57.583420 1189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:57.582520 1192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:57.584769 1183 server_base.cc:1034] running on GCE node
I20260929 03:12:57.585345 1183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:57.586208 1183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:57.587432 1183 hybrid_clock.cc:648] HybridClock initialized: now 1790651577587350 us; error 119 us; skew 500 ppm
Sep 29 03:12:57 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651577, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:57.591491 1183 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:12:57.592685 1183 webserver.cc:460] Webserver started at http://127.20.174.66:46739/ using document root <none> and password file <none>
I20260929 03:12:57.593425 1183 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:57.593520 1183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:12:57.593755 1183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:12:57 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651577, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:57.617022 1183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "142b709315ac44ec932db284df70dacc"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "1b7b6d51157f0f9337d88ef10b338bda"
server_key_iv: "79e89b5114f0ef88b94f9aafa84dcde3"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:57.617821 1183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "142b709315ac44ec932db284df70dacc"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "1b7b6d51157f0f9337d88ef10b338bda"
server_key_iv: "79e89b5114f0ef88b94f9aafa84dcde3"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:57.623207 1183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 03:12:57.648522 1199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:57.650403 1183 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260929 03:12:57.650566 1183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "142b709315ac44ec932db284df70dacc"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "1b7b6d51157f0f9337d88ef10b338bda"
server_key_iv: "79e89b5114f0ef88b94f9aafa84dcde3"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:57.650707 1183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:57.665612 1183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:57.673630 1183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:57.673918 1183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:57.674746 1183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:12:57.675957 1183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:12:57.676071 1183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:57.676173 1183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:12:57.676236 1183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:57.698559 1183 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:42801
I20260929 03:12:57.699965 1183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:12:57.704141 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 1183
I20260929 03:12:57.704279 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:12:57.704641 21177 external_mini_cluster.cc:1428] Setting key 3151477b3f5525b91df2a4db2119a1f0
I20260929 03:12:57.704787 1312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:42801 every 8 connection(s)
Sep 29 03:12:57 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651577, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:12:57.719913 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:57.704239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:35553 (local address 127.20.174.126:39777)
0929 03:12:57.705207 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:57.705213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:57.705549 (+ 336us) server_negotiation.cc:407] Connection header received
0929 03:12:57.707079 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:57.707083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:57.707134 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:57.707187 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:57.708841 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:57.709632 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:57.711502 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:57.711869 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:57.711946 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:57.715140 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:57.715145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:57.715148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:57.715178 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:57.717088 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:57.717809 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:57.717813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:57.717815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:57.717915 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:57.718388 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:57.718392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:57.718394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:57.718677 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:57.718775 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:57.719570 (+ 795us) server_negotiation.cc:299] Negotiation successful
0929 03:12:57.719727 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862}
I20260929 03:12:57.722141 1315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:57.703117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39777 (local address 127.20.174.66:35553)
0929 03:12:57.705378 (+ 2261us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:57.705437 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:57.706581 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:57.707609 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:57.707618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:57.708005 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:12:57.708668 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:57.708689 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:57.709806 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:57.709809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:57.711339 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:57.711363 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:57.712153 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:57.712157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:57.712377 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:57.713083 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:12:57.713104 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:12:57.714912 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:12:57.717242 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:57.717249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:57.717268 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:57.717674 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:57.718039 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:57.718043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:57.718045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:57.718250 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:57.718857 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:12:57.718864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:12:57.719230 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 03:12:57.720440 (+ 1210us) client_negotiation.cc:240] Negotiation successful
0929 03:12:57.720849 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2009,"thread_start_us":140,"threads_started":1}
I20260929 03:12:57.724145 1313 heartbeater.cc:347] Connected to a master server at 127.20.174.126:39777
I20260929 03:12:57.724445 1313 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:57.725081 1313 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
I20260929 03:12:57.726491 956 ts_manager.cc:194] Registered new tserver with Master: 142b709315ac44ec932db284df70dacc (127.20.174.66:42801)
I20260929 03:12:57.727731 956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:35553
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:12:57.757961 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:39679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39777
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:12:57.914309 1318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:57.914728 1318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:57.914790 1318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:57.920989 1318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:57.921126 1318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:57.921228 1318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:12:57.929596 1318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.174.84:39679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:39777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:57.931057 1318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:57.932479 1318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:12:57.945928 1326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:57.945958 1323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:57.946125 1324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:57.947523 1318 server_base.cc:1034] running on GCE node
I20260929 03:12:57.948068 1318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:57.948892 1318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:57.950083 1318 hybrid_clock.cc:648] HybridClock initialized: now 1790651577950054 us; error 40 us; skew 500 ppm
Sep 29 03:12:57 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651577, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:57.953946 1318 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:12:57.955224 1318 webserver.cc:460] Webserver started at http://127.20.174.67:36929/ using document root <none> and password file <none>
I20260929 03:12:57.956000 1318 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:57.956090 1318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:12:57.956350 1318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:12:57 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651577, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
Sep 29 03:12:57 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651577, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:57.984078 1318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "45907cb081ae4faf8bd004d3fe3ea34d"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "ebd2aa294106aa7c0372905a7fa5064f"
server_key_iv: "2db82818c494e95f9787b27bfe80538c"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:57.984975 1318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "45907cb081ae4faf8bd004d3fe3ea34d"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "ebd2aa294106aa7c0372905a7fa5064f"
server_key_iv: "2db82818c494e95f9787b27bfe80538c"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:57.990257 1318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 03:12:58.010326 1333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:58.016330 1318 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.000s
I20260929 03:12:58.016607 1318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "45907cb081ae4faf8bd004d3fe3ea34d"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "ebd2aa294106aa7c0372905a7fa5064f"
server_key_iv: "2db82818c494e95f9787b27bfe80538c"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:58.016875 1318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:58.059185 1318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:58.063571 1318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:58.063851 1318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:58.064674 1318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:12:58.065933 1318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:12:58.066049 1318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:58.066154 1318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:12:58.066218 1318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:58.080444 1318 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:37039
I20260929 03:12:58.081758 1318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:12:58.082652 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 1318
I20260929 03:12:58.082785 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:12:58.083096 21177 external_mini_cluster.cc:1428] Setting key c1f880036b2c80562958ba70558f2c65
I20260929 03:12:58.086513 1446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:37039 every 8 connection(s)
Sep 29 03:12:58 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651577, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:12:58.106577 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:58.090711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:41535 (local address 127.20.174.126:39777)
0929 03:12:58.090835 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:58.090840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:58.091565 (+ 725us) server_negotiation.cc:407] Connection header received
0929 03:12:58.092816 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:58.092820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:58.092874 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:58.092925 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:58.095356 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.096223 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.098129 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.098657 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.098750 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:58.102182 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:58.102186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:58.102189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:58.102219 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:58.104070 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:58.104734 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:58.104737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:58.104739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:58.104828 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:58.105219 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:58.105222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:58.105224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:58.105454 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:58.105525 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:58.106225 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 03:12:58.106379 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:12:58.108587 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:58.089527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39777 (local address 127.20.174.67:41535)
0929 03:12:58.091401 (+ 1874us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:58.091461 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:58.092563 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:58.093353 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:58.093362 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:58.093729 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:12:58.095118 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:58.095137 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.096454 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.096458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:58.097965 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:58.097979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.099077 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.099083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:58.099324 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:58.100158 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:12:58.100181 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:12:58.101979 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:12:58.104223 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:58.104229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:58.104262 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:58.104617 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:58.104935 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:58.104938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:58.104940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:58.105116 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:58.105606 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:12:58.105613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:12:58.105948 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 03:12:58.107048 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0929 03:12:58.107439 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1626,"thread_start_us":149,"threads_started":1}
I20260929 03:12:58.110466 1447 heartbeater.cc:347] Connected to a master server at 127.20.174.126:39777
I20260929 03:12:58.110735 1447 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:58.111357 1447 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
I20260929 03:12:58.112723 963 ts_manager.cc:194] Registered new tserver with Master: 45907cb081ae4faf8bd004d3fe3ea34d (127.20.174.67:37039)
I20260929 03:12:58.113472 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:41535
I20260929 03:12:58.125588 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:12:58.126353 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 1044
I20260929 03:12:58.135690 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 1183
I20260929 03:12:58.144644 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 1318
I20260929 03:12:58.153074 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 931
I20260929 03:12:58.161465 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:39777
--webserver_interface=127.20.174.126
--webserver_port=41103
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:39777
--ranger_config_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:39679/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.174.84:39679/kms with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:12:58.300861 1450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:58.301160 1450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:58.301204 1450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:58.305487 1450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:12:58.305562 1450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:58.305583 1450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:58.305599 1450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:12:58.305614 1450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:12:58.311525 1450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.174.84:39679/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:39777
--ranger_config_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:39777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=41103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:58.312724 1450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:58.313723 1450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:12:58.322690 1455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:58.322674 1456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:58.322679 1458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:58.322991 1450 server_base.cc:1034] running on GCE node
I20260929 03:12:58.323717 1450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:58.324856 1450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:58.326063 1450 hybrid_clock.cc:648] HybridClock initialized: now 1790651578326047 us; error 45 us; skew 500 ppm
Sep 29 03:12:58 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651578, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:58.329288 1450 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:12:58.330374 1450 webserver.cc:460] Webserver started at http://127.20.174.126:41103/ using document root <none> and password file <none>
I20260929 03:12:58.331019 1450 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:58.331130 1450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:12:58 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651578, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:58.401043 1450 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:12:58.403690 1466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:58.405114 1450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:12:58.405282 1450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "a1cb3d743c38483ca25a22b30c5ea349"
format_stamp: "Formatted at 2026-09-29 03:12:56 on dist-test-slave-k1dg"
server_key: "86435eff21ad2fbfc9486d41a7a6cdd5"
server_key_iv: "2cfbbac915ecf24e3c1c248c641cee38"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:58.405856 1450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:58.440603 1450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:58.443583 1450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:58.443820 1450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:58.451982 1450 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:39777
I20260929 03:12:58.452016 1518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:39777 every 8 connection(s)
I20260929 03:12:58.453115 1450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:12:58.457536 1519 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:12:58.459349 1519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349: Bootstrap starting.
I20260929 03:12:58.461257 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 1450
I20260929 03:12:58.461716 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:39679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:43601
--local_ip_for_outbound_sockets=127.20.174.65
--tserver_master_addrs=127.20.174.126:39777
--webserver_port=33741
--webserver_interface=127.20.174.65
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.174.84:39679/kms with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:12:58.476790 1519 log.cc:824] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349: Log is configured to *not* fsync() on all Append() calls
I20260929 03:12:58.484319 1519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:12:58.484774 1519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349: Bootstrap complete.
I20260929 03:12:58.489153 1519 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } }
I20260929 03:12:58.489578 1519 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1cb3d743c38483ca25a22b30c5ea349, State: Initialized, Role: FOLLOWER
I20260929 03:12:58.490098 1519 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } }
I20260929 03:12:58.490217 1519 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:12:58.490311 1519 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:12:58.490411 1519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:12:58.492481 1519 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } }
I20260929 03:12:58.492851 1519 leader_election.cc:302] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1cb3d743c38483ca25a22b30c5ea349; no voters:
I20260929 03:12:58.493193 1519 leader_election.cc:288] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:12:58.493342 1523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:12:58.493593 1523 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [term 2 LEADER]: Becoming Leader. State: Replica: a1cb3d743c38483ca25a22b30c5ea349, State: Running, Role: LEADER
I20260929 03:12:58.493911 1523 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } }
I20260929 03:12:58.494535 1519 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:12:58.495502 1525 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1cb3d743c38483ca25a22b30c5ea349. Latest consensus state: current_term: 2 leader_uuid: "a1cb3d743c38483ca25a22b30c5ea349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } } }
I20260929 03:12:58.495544 1524 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a1cb3d743c38483ca25a22b30c5ea349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb3d743c38483ca25a22b30c5ea349" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39777 } } }
I20260929 03:12:58.495672 1524 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [sys.catalog]: This master's current role is: LEADER
I20260929 03:12:58.495671 1525 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349 [sys.catalog]: This master's current role is: LEADER
I20260929 03:12:58.496150 1532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:12:58.499501 1532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:12:58.500089 1519 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:12:58.500125 1532 catalog_manager.cc:1259] Loaded cluster ID: 62ab49d81d2c47478159b5473f674586
I20260929 03:12:58.500175 1532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:12:58.502503 1532 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:12:58.503207 1532 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349: Loaded TSK: 0
I20260929 03:12:58.504436 1532 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:12:58.585393 1521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:58.585690 1521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:58.585780 1521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:58.590596 1521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:58.590680 1521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:58.590786 1521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:12:58.597425 1521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.174.84:39679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:43601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=33741
--tserver_master_addrs=127.20.174.126:39777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:58.598821 1521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:58.600045 1521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:12:58.610270 1557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:58.610457 1556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:58.610579 1559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:58.613215 1521 server_base.cc:1034] running on GCE node
I20260929 03:12:58.613726 1521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:58.614573 1521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:58.615777 1521 hybrid_clock.cc:648] HybridClock initialized: now 1790651578615738 us; error 59 us; skew 500 ppm
Sep 29 03:12:58 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651578, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:58.619575 1521 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:12:58.620790 1521 webserver.cc:460] Webserver started at http://127.20.174.65:33741/ using document root <none> and password file <none>
I20260929 03:12:58.621587 1521 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:58.621695 1521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:12:58 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651578, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:58.652385 1521 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:12:58.655601 1567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:58.657208 1521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:12:58.657356 1521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "0c35421495e049d1b0b4487f06cc63ab"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "69ba1a6e50f82dfe246877fc205effa4"
server_key_iv: "3dedb4133d200253671f64d8044f8cde"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:58.658113 1521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:58.697723 1521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:58.701668 1521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:58.701881 1521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:58.702559 1521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:12:58.703603 1521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:12:58.703665 1521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:58.703727 1521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:12:58.703756 1521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:58.713883 1521 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:43601
I20260929 03:12:58.715292 1521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:12:58.718638 1680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:43601 every 8 connection(s)
I20260929 03:12:58.720587 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 1521
I20260929 03:12:58.721082 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:39679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:42801
--local_ip_for_outbound_sockets=127.20.174.66
--tserver_master_addrs=127.20.174.126:39777
--webserver_port=46739
--webserver_interface=127.20.174.66
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.174.84:39679/kms with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:12:58 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651578, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:12:58.743192 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:58.721119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39777 (local address 127.20.174.65:40903)
0929 03:12:58.724209 (+ 3090us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:58.724272 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:58.725571 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:58.727931 (+ 2360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:58.727944 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:58.728347 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:12:58.729089 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:58.729116 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.731326 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.731332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:58.732962 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:58.732976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.734379 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.734383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:58.734596 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:58.735331 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:12:58.735353 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:12:58.736898 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:12:58.739433 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:58.739439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:58.739454 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:58.739790 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:58.740178 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:58.740181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:58.740183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:58.740353 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:58.741120 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:12:58.741126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:12:58.741446 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 03:12:58.741651 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 03:12:58.742031 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2835,"thread_start_us":149,"threads_started":1}
I20260929 03:12:58.745333 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:58.722748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:40903 (local address 127.20.174.126:39777)
0929 03:12:58.723318 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:58.723346 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:58.725955 (+ 2609us) server_negotiation.cc:407] Connection header received
0929 03:12:58.726335 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:58.726354 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:58.726879 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:58.727290 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:58.729338 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.731129 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.733131 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.733982 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.734140 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:58.737136 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:58.737142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:58.737159 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:58.737191 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:58.739278 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:58.739928 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:58.739935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:58.739941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:58.740080 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:58.740456 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:58.740459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:58.740461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:58.740943 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:58.741058 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:58.743925 (+ 2867us) server_negotiation.cc:299] Negotiation successful
0929 03:12:58.744275 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":124,"threads_started":1}
I20260929 03:12:58.748219 1681 heartbeater.cc:347] Connected to a master server at 127.20.174.126:39777
I20260929 03:12:58.748514 1681 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:58.749135 1681 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
W20260929 03:12:58.751554 1681 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:39777 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:12:58.751689 1681 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:58.751875 1681 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
I20260929 03:12:58.752820 1681 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:58.752996 1681 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
W20260929 03:12:58.753731 1681 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:12:58.869710 1684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:58.870107 1684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:58.870182 1684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:58.875823 1684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:58.875917 1684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:58.876041 1684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:12:58.883267 1684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.174.84:39679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:42801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=46739
--tserver_master_addrs=127.20.174.126:39777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:58.884665 1684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:58.886066 1684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:12:58.897962 1690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:58.897962 1691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:58.898411 1693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:58.899029 1684 server_base.cc:1034] running on GCE node
I20260929 03:12:58.899561 1684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:58.900388 1684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:58.901603 1684 hybrid_clock.cc:648] HybridClock initialized: now 1790651578901584 us; error 72 us; skew 500 ppm
Sep 29 03:12:58 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651578, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:58.904711 1684 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:12:58.905797 1684 webserver.cc:460] Webserver started at http://127.20.174.66:46739/ using document root <none> and password file <none>
I20260929 03:12:58.906417 1684 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:58.906590 1684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:12:58 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651578, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:58.929692 1684 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:12:58.935375 1701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:58.937177 1684 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260929 03:12:58.937332 1684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "142b709315ac44ec932db284df70dacc"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "1b7b6d51157f0f9337d88ef10b338bda"
server_key_iv: "79e89b5114f0ef88b94f9aafa84dcde3"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:58.938128 1684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:58.956038 1684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:58.959852 1684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:58.960078 1684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:58.960853 1684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:12:58.961957 1684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:12:58.962020 1684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:58.962085 1684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:12:58.962116 1684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:58.975798 1684 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:42801
I20260929 03:12:58.975988 1814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:42801 every 8 connection(s)
I20260929 03:12:58.976951 1684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:12:58.981479 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 1684
I20260929 03:12:58.982009 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.174.84:39679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:37039
--local_ip_for_outbound_sockets=127.20.174.67
--tserver_master_addrs=127.20.174.126:39777
--webserver_port=36929
--webserver_interface=127.20.174.67
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.174.84:39679/kms with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:12:58 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651578, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:12:58.998679 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:58.980658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:57737 (local address 127.20.174.126:39777)
0929 03:12:58.980850 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:58.980857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:58.981197 (+ 340us) server_negotiation.cc:407] Connection header received
0929 03:12:58.982233 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:58.982236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:58.982301 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:58.982392 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:58.983925 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.984765 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.986830 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.987204 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.987321 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:58.993165 (+ 5844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:58.993170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:58.993173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:58.993213 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:58.995334 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:58.996435 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:58.996440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:58.996442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:58.996545 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:58.997015 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:58.997019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:58.997021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:58.997266 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:58.997538 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:58.998300 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 03:12:58.998491 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:12:58.999740 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:58.979556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39777 (local address 127.20.174.66:57737)
0929 03:12:58.981014 (+ 1458us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:58.981077 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:58.982002 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:58.982763 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:58.982771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:58.983062 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:12:58.983716 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:58.983742 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.984939 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.984946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:58.986661 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:58.986679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:58.987540 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:58.987545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:58.987775 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:58.990899 (+ 3124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:12:58.990923 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:12:58.992893 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:12:58.995487 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:58.995495 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:58.995514 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:58.996191 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:58.996658 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:58.996661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:58.996663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:58.996875 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:58.997604 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:12:58.997611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:12:58.997975 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 03:12:58.998209 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 03:12:58.998617 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"thread_start_us":142,"threads_started":1}
I20260929 03:12:59.001195 1815 heartbeater.cc:347] Connected to a master server at 127.20.174.126:39777
I20260929 03:12:59.001469 1815 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:59.002102 1815 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
W20260929 03:12:59.004043 1815 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:39777 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:12:59.004262 1815 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:59.004508 1815 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
I20260929 03:12:59.005645 1815 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:59.005875 1815 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
W20260929 03:12:59.007002 1815 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:12:59.127748 1818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:12:59.128110 1818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:12:59.128201 1818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:12:59.133419 1818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:12:59.133527 1818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:12:59.133625 1818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:12:59.141245 1818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.174.84:39679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:37039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=36929
--tserver_master_addrs=127.20.174.126:39777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:12:59.142768 1818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:12:59.143992 1818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:12:59.154362 1824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:59.154855 1825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:12:59.154362 1827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:12:59.155606 1818 server_base.cc:1034] running on GCE node
I20260929 03:12:59.156301 1818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:12:59.157193 1818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:12:59.158416 1818 hybrid_clock.cc:648] HybridClock initialized: now 1790651579158395 us; error 39 us; skew 500 ppm
Sep 29 03:12:59 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651579, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:59.162175 1818 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:12:59.163417 1818 webserver.cc:460] Webserver started at http://127.20.174.67:36929/ using document root <none> and password file <none>
I20260929 03:12:59.164245 1818 fs_manager.cc:359] Metadata directory not provided
I20260929 03:12:59.164359 1818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:12:59 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651579, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:12:59.185607 1818 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:12:59.189183 1835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:59.190932 1818 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 03:12:59.191075 1818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "45907cb081ae4faf8bd004d3fe3ea34d"
format_stamp: "Formatted at 2026-09-29 03:12:57 on dist-test-slave-k1dg"
server_key: "ebd2aa294106aa7c0372905a7fa5064f"
server_key_iv: "2db82818c494e95f9787b27bfe80538c"
server_key_version: "kuduclusterkey@0"
I20260929 03:12:59.191852 1818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:12:59.206110 1818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:12:59.209815 1818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:12:59.209982 1818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:12:59.210731 1818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:12:59.211714 1818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:12:59.211769 1818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:59.211836 1818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:12:59.211869 1818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:12:59.223232 1818 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:37039
I20260929 03:12:59.224542 1818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:12:59.225497 1948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:37039 every 8 connection(s)
I20260929 03:12:59.232736 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 1818
Sep 29 03:12:59 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651579, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:12:59.243252 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:59.227791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:43655 (local address 127.20.174.126:39777)
0929 03:12:59.227940 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:59.227944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:59.229581 (+ 1637us) server_negotiation.cc:407] Connection header received
0929 03:12:59.230945 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:59.230950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:59.231018 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:59.231134 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:59.232723 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:59.233459 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:59.235139 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:59.235379 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:59.235540 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:59.238243 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:59.238247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:59.238251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:59.238303 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:59.240372 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:59.241135 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:59.241139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:59.241142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:59.241231 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:59.241736 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:59.241743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:59.241747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:59.241992 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:59.242067 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:59.242916 (+ 849us) server_negotiation.cc:299] Negotiation successful
0929 03:12:59.243061 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56,"spinlock_wait_cycles":48256}
I20260929 03:12:59.244777 1953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:59.233878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.20.174.126:39777)
0929 03:12:59.234566 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 03:12:59.234575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:12:59.234587 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:12:59.234622 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:12:59.234625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:12:59.234666 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:12:59.234754 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:12:59.235760 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:59.236489 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:59.237730 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:12:59.238094 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:12:59.238210 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:59.240583 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:12:59.240586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:12:59.240589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:12:59.240614 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:12:59.242355 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:59.242870 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:59.242873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:59.242874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:59.242967 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:12:59.243265 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:12:59.243271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:12:59.243276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:12:59.243570 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:12:59.243759 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 03:12:59.244504 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 03:12:59.244622 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"spinlock_wait_cycles":4608,"thread_start_us":98,"threads_started":1}
I20260929 03:12:59.245496 1951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:12:59.228388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39777 (local address 127.20.174.67:43655)
0929 03:12:59.229395 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0929 03:12:59.229457 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 03:12:59.230649 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:12:59.231364 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:12:59.231375 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:12:59.231755 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:12:59.232509 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:59.232532 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:59.233623 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:59.233627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:59.235014 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:12:59.235023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:12:59.235594 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:12:59.235598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:12:59.235822 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:12:59.236541 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:12:59.236569 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:12:59.238054 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:12:59.240541 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:59.240548 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:59.240570 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:59.241007 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:59.241363 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:12:59.241368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:12:59.241372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:12:59.241608 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:12:59.242159 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:12:59.242166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:12:59.242594 (+ 428us) client_negotiation.cc:769] Sending connection context
0929 03:12:59.243697 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0929 03:12:59.244082 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":166,"threads_started":1}
I20260929 03:12:59.246794 1949 heartbeater.cc:347] Connected to a master server at 127.20.174.126:39777
I20260929 03:12:59.247058 1949 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:59.247579 1949 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
W20260929 03:12:59.249140 1949 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:39777 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:12:59.249279 1949 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:59.249455 1949 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
I20260929 03:12:59.250324 1949 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:59.250523 1949 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
W20260929 03:12:59.252240 1949 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 03:12:59 dist-test-slave-k1dg krb5kdc[487](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651579, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:12:59.753957 1681 heartbeater.cc:464] Registering TS with master...
I20260929 03:12:59.754256 1681 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
I20260929 03:12:59.755462 1482 ts_manager.cc:194] Registered new tserver with Master: 0c35421495e049d1b0b4487f06cc63ab (127.20.174.65:43601)
I20260929 03:12:59.757243 1482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:40903
I20260929 03:13:00.007270 1815 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:00.007604 1815 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
I20260929 03:13:00.008390 1482 ts_manager.cc:194] Registered new tserver with Master: 142b709315ac44ec932db284df70dacc (127.20.174.66:42801)
I20260929 03:13:00.009093 1482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:57737
Sep 29 03:13:00 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651579, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:13:00.252472 1949 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:00.252763 1949 heartbeater.cc:511] Master 127.20.174.126:39777 requested a full tablet report, sending...
I20260929 03:13:00.253542 1482 ts_manager.cc:194] Registered new tserver with Master: 45907cb081ae4faf8bd004d3fe3ea34d (127.20.174.67:37039)
I20260929 03:13:00.254359 1482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:43655
I20260929 03:13:00.264798 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:13:00.278201 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.267795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.20.174.126:39777)
0929 03:13:00.268374 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:00.268379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:00.268696 (+ 317us) server_negotiation.cc:407] Connection header received
0929 03:13:00.268737 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:00.268740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:00.268788 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:00.268866 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:00.270131 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.271021 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.272138 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.272476 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.272564 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.274315 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:00.274320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:00.274323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:00.274355 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:00.276306 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:00.276817 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:00.276821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:00.276823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:00.276925 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:00.277229 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:00.277233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:00.277235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:00.277467 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:00.277605 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:00.277858 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 03:13:00.278001 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":85,"threads_started":1}
I20260929 03:13:00.283046 1482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:13:00.288271 1482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:13:00.310122 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.301839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:42801 (local address 127.0.0.1:39054)
0929 03:13:00.303480 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:00.303545 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:00.303692 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:00.304924 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:00.304946 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:00.304973 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:00.305128 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:00.305136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.306629 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.306632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:00.308073 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:00.308086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.309472 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.309476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:00.309588 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.309625 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:13:00.309803 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 03:13:00.309940 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1436,"thread_start_us":112,"threads_started":1}
I20260929 03:13:00.310910 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.303137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:37039 (local address 127.0.0.1:37484)
0929 03:13:00.303889 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:00.303902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:00.303992 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:00.305474 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:00.305477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:00.305494 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:00.305577 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:00.305582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.306943 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.306946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:00.308970 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:00.308982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.310460 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.310464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:00.310596 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.310619 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:13:00.310684 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:13:00.310764 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"spinlock_wait_cycles":6400,"thread_start_us":96,"threads_started":1}
I20260929 03:13:00.311300 1980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.302673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.20.174.66:42801)
0929 03:13:00.304230 (+ 1557us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:00.304238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:00.304261 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:13:00.304341 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:00.304346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:00.304569 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:00.304732 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:00.305283 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.306499 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.308231 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.309154 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.309249 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.309410 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:00.310829 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0929 03:13:00.310953 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1456,"thread_start_us":107,"threads_started":1}
I20260929 03:13:00.311458 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.303526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.20.174.67:37039)
0929 03:13:00.304229 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:00.304238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:00.304261 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:13:00.304341 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:00.304346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:00.304569 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:00.305310 (+ 741us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:00.305709 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.306820 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.309154 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.310027 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.310225 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.310392 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:00.311198 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 03:13:00.311304 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":95,"threads_started":1}
I20260929 03:13:00.314548 1749 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1e6258df274250a59b3bbdcbe510a6 (DEFAULT_TABLE table=test-table [id=991975ef1e354479b8c00f80a0204076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:00.314553 1865 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1e6258df274250a59b3bbdcbe510a6 (DEFAULT_TABLE table=test-table [id=991975ef1e354479b8c00f80a0204076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:00.315812 1749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1e6258df274250a59b3bbdcbe510a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:00.316630 1865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1e6258df274250a59b3bbdcbe510a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:00.321522 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.301953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:43601 (local address 127.0.0.1:41868)
0929 03:13:00.303891 (+ 1938us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:00.303904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:00.303992 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:00.311424 (+ 7432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:00.311428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:00.311448 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:00.311548 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:00.311554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.318555 (+ 7001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.318559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:00.319823 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:00.319837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.321053 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.321057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:00.321156 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.321189 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:13:00.321250 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:13:00.321347 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1872,"spinlock_wait_cycles":5760,"thread_start_us":77,"threads_started":1}
I20260929 03:13:00.322599 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.302121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.20.174.65:43601)
0929 03:13:00.310684 (+ 8563us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:00.310691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:00.310720 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:13:00.310817 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:00.310823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:00.311075 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:00.311250 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:00.311693 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.312813 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.320020 (+ 7207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.320894 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.322081 (+ 1187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.322265 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:00.322323 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:13:00.322449 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8431,"thread_start_us":97,"threads_started":1}
I20260929 03:13:00.325814 1615 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1e6258df274250a59b3bbdcbe510a6 (DEFAULT_TABLE table=test-table [id=991975ef1e354479b8c00f80a0204076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:00.326973 1615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1e6258df274250a59b3bbdcbe510a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:00.335058 1983 tablet_bootstrap.cc:492] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: Bootstrap starting.
I20260929 03:13:00.337713 1984 tablet_bootstrap.cc:492] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: Bootstrap starting.
I20260929 03:13:00.337759 1983 tablet_bootstrap.cc:654] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: Neither blocks nor log segments found. Creating new log.
Sep 29 03:13:00 dist-test-slave-k1dg krb5kdc[487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651579, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for HTTP/127.20.174.84@KRBTEST.COM
I20260929 03:13:00.339918 1983 log.cc:824] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:00.340852 1985 tablet_bootstrap.cc:492] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab: Bootstrap starting.
I20260929 03:13:00.344055 1985 tablet_bootstrap.cc:654] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:00.345103 1985 log.cc:824] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:00.346220 1983 tablet_bootstrap.cc:492] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: No bootstrap required, opened a new log
I20260929 03:13:00.346549 1983 ts_tablet_manager.cc:1397] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260929 03:13:00.350946 1983 raft_consensus.cc:348] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b709315ac44ec932db284df70dacc" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 42801 } } peers { permanent_uuid: "0c35421495e049d1b0b4487f06cc63ab" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43601 } } peers { permanent_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 37039 } }
I20260929 03:13:00.351351 1983 raft_consensus.cc:374] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:00.351440 1983 raft_consensus.cc:729] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142b709315ac44ec932db284df70dacc, State: Initialized, Role: FOLLOWER
I20260929 03:13:00.352047 1983 consensus_queue.cc:260] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b709315ac44ec932db284df70dacc" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 42801 } } peers { permanent_uuid: "0c35421495e049d1b0b4487f06cc63ab" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43601 } } peers { permanent_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 37039 } }
I20260929 03:13:00.353329 1983 ts_tablet_manager.cc:1428] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 03:13:00.353781 1815 heartbeater.cc:503] Master 127.20.174.126:39777 was elected leader, sending a full tablet report...
I20260929 03:13:00.356945 1984 tablet_bootstrap.cc:654] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:00.358002 1984 log.cc:824] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:00.360196 1985 tablet_bootstrap.cc:492] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab: No bootstrap required, opened a new log
I20260929 03:13:00.360483 1985 ts_tablet_manager.cc:1397] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260929 03:13:00.362712 1984 tablet_bootstrap.cc:492] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: No bootstrap required, opened a new log
I20260929 03:13:00.362931 1984 ts_tablet_manager.cc:1397] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.002s
I20260929 03:13:00.366245 1985 raft_consensus.cc:348] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b709315ac44ec932db284df70dacc" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 42801 } } peers { permanent_uuid: "0c35421495e049d1b0b4487f06cc63ab" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43601 } } peers { permanent_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 37039 } }
I20260929 03:13:00.366670 1985 raft_consensus.cc:374] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:00.366791 1985 raft_consensus.cc:729] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c35421495e049d1b0b4487f06cc63ab, State: Initialized, Role: FOLLOWER
I20260929 03:13:00.367396 1985 consensus_queue.cc:260] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b709315ac44ec932db284df70dacc" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 42801 } } peers { permanent_uuid: "0c35421495e049d1b0b4487f06cc63ab" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43601 } } peers { permanent_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 37039 } }
I20260929 03:13:00.367369 1984 raft_consensus.cc:348] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b709315ac44ec932db284df70dacc" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 42801 } } peers { permanent_uuid: "0c35421495e049d1b0b4487f06cc63ab" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43601 } } peers { permanent_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 37039 } }
I20260929 03:13:00.367632 1984 raft_consensus.cc:374] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:00.367722 1984 raft_consensus.cc:729] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45907cb081ae4faf8bd004d3fe3ea34d, State: Initialized, Role: FOLLOWER
I20260929 03:13:00.368219 1984 consensus_queue.cc:260] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b709315ac44ec932db284df70dacc" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 42801 } } peers { permanent_uuid: "0c35421495e049d1b0b4487f06cc63ab" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43601 } } peers { permanent_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 37039 } }
I20260929 03:13:00.368868 1985 ts_tablet_manager.cc:1428] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 03:13:00.369171 1984 ts_tablet_manager.cc:1428] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260929 03:13:00.369290 1681 heartbeater.cc:503] Master 127.20.174.126:39777 was elected leader, sending a full tablet report...
I20260929 03:13:00.369526 1949 heartbeater.cc:503] Master 127.20.174.126:39777 was elected leader, sending a full tablet report...
W20260929 03:13:00.472579 1682 tablet.cc:2369] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:13:00.477263 1950 tablet.cc:2369] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:13:00.478986 1816 tablet.cc:2369] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:13:00.557065 1988 raft_consensus.cc:482] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:13:00.557400 1988 raft_consensus.cc:504] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b709315ac44ec932db284df70dacc" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 42801 } } peers { permanent_uuid: "0c35421495e049d1b0b4487f06cc63ab" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43601 } } peers { permanent_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 37039 } }
I20260929 03:13:00.559001 1988 leader_election.cc:288] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c35421495e049d1b0b4487f06cc63ab (127.20.174.65:43601), 45907cb081ae4faf8bd004d3fe3ea34d (127.20.174.67:37039)
I20260929 03:13:00.565438 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.559947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:42117 (local address 127.20.174.67:37039)
0929 03:13:00.560104 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:00.560113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:00.560417 (+ 304us) server_negotiation.cc:407] Connection header received
0929 03:13:00.560625 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:00.560629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:00.560698 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:00.560752 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:00.561179 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.561951 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.563755 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.564422 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.564553 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.564613 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:00.565215 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 03:13:00.565291 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:13:00.565438 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.559754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:46787 (local address 127.20.174.65:43601)
0929 03:13:00.559921 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:00.559927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:00.559953 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:13:00.560200 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:00.560204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:00.560262 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:00.560323 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:00.560833 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.561742 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.563747 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.564432 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.564553 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.564613 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:00.565192 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 03:13:00.565283 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:13:00.565979 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.559227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:43601 (local address 127.20.174.66:46787)
0929 03:13:00.559867 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:00.559888 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:00.560020 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:00.560496 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:00.560500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:00.560516 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:00.560679 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:00.560688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.561932 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.561935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:00.563568 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:00.563581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.564789 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.564793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:00.565004 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.565037 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:13:00.565602 (+ 565us) client_negotiation.cc:240] Negotiation successful
0929 03:13:00.565781 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":130,"thread_start_us":147,"threads_started":1}
I20260929 03:13:00.566383 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.559336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:37039 (local address 127.20.174.66:42117)
0929 03:13:00.560322 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:00.560337 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:00.560493 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:00.560904 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:00.560907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:00.560923 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:00.561041 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:00.561048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.562083 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.562086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:00.563568 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:00.563581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.564789 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.564793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:00.565056 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.565089 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:13:00.566038 (+ 949us) client_negotiation.cc:240] Negotiation successful
0929 03:13:00.566175 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":98,"thread_start_us":77,"threads_started":1}
I20260929 03:13:00.566890 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1e6258df274250a59b3bbdcbe510a6" candidate_uuid: "142b709315ac44ec932db284df70dacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c35421495e049d1b0b4487f06cc63ab" is_pre_election: true
I20260929 03:13:00.567287 1635 raft_consensus.cc:2454] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142b709315ac44ec932db284df70dacc in term 0.
I20260929 03:13:00.567855 1885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1e6258df274250a59b3bbdcbe510a6" candidate_uuid: "142b709315ac44ec932db284df70dacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" is_pre_election: true
I20260929 03:13:00.567929 1702 leader_election.cc:302] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c35421495e049d1b0b4487f06cc63ab, 142b709315ac44ec932db284df70dacc; no voters:
I20260929 03:13:00.568121 1885 raft_consensus.cc:2454] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142b709315ac44ec932db284df70dacc in term 0.
I20260929 03:13:00.568300 1988 raft_consensus.cc:2781] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:13:00.568387 1988 raft_consensus.cc:482] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:13:00.568434 1988 raft_consensus.cc:3037] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:00.569983 1988 raft_consensus.cc:504] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b709315ac44ec932db284df70dacc" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 42801 } } peers { permanent_uuid: "0c35421495e049d1b0b4487f06cc63ab" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43601 } } peers { permanent_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 37039 } }
I20260929 03:13:00.570463 1988 leader_election.cc:288] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [CANDIDATE]: Term 1 election: Requested vote from peers 0c35421495e049d1b0b4487f06cc63ab (127.20.174.65:43601), 45907cb081ae4faf8bd004d3fe3ea34d (127.20.174.67:37039)
I20260929 03:13:00.570930 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1e6258df274250a59b3bbdcbe510a6" candidate_uuid: "142b709315ac44ec932db284df70dacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c35421495e049d1b0b4487f06cc63ab"
I20260929 03:13:00.571105 1885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1e6258df274250a59b3bbdcbe510a6" candidate_uuid: "142b709315ac44ec932db284df70dacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45907cb081ae4faf8bd004d3fe3ea34d"
I20260929 03:13:00.571136 1635 raft_consensus.cc:3037] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:00.571228 1885 raft_consensus.cc:3037] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:00.572606 1635 raft_consensus.cc:2454] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142b709315ac44ec932db284df70dacc in term 1.
I20260929 03:13:00.572625 1885 raft_consensus.cc:2454] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142b709315ac44ec932db284df70dacc in term 1.
I20260929 03:13:00.572983 1703 leader_election.cc:302] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142b709315ac44ec932db284df70dacc, 45907cb081ae4faf8bd004d3fe3ea34d; no voters:
I20260929 03:13:00.573390 1988 raft_consensus.cc:2781] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:13:00.573659 1988 raft_consensus.cc:686] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 1 LEADER]: Becoming Leader. State: Replica: 142b709315ac44ec932db284df70dacc, State: Running, Role: LEADER
I20260929 03:13:00.574065 1988 consensus_queue.cc:237] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b709315ac44ec932db284df70dacc" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 42801 } } peers { permanent_uuid: "0c35421495e049d1b0b4487f06cc63ab" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43601 } } peers { permanent_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 37039 } }
I20260929 03:13:00.578176 1482 catalog_manager.cc:5495] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc reported cstate change: term changed from 0 to 1, leader changed from <none> to 142b709315ac44ec932db284df70dacc (127.20.174.66). New cstate: current_term: 1 leader_uuid: "142b709315ac44ec932db284df70dacc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b709315ac44ec932db284df70dacc" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 42801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c35421495e049d1b0b4487f06cc63ab" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 37039 } health_report { overall_health: UNKNOWN } } }
I20260929 03:13:00.603785 1980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.598348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.20.174.66:42801)
0929 03:13:00.598491 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:00.598496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:00.598509 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:13:00.598599 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:00.598602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:00.598667 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:00.598791 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:13:00.599183 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.599940 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.601245 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:00.601692 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:00.601830 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:00.602230 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:13:00.602905 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:13:00.603033 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:00.603525 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 03:13:00.603597 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:13:00.612373 1885 raft_consensus.cc:1261] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d [term 1 FOLLOWER]: Refusing update from remote peer 142b709315ac44ec932db284df70dacc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:00.612373 1635 raft_consensus.cc:1261] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab [term 1 FOLLOWER]: Refusing update from remote peer 142b709315ac44ec932db284df70dacc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:00.613183 1988 consensus_queue.cc:1035] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [LEADER]: Connected to new peer: Peer: permanent_uuid: "45907cb081ae4faf8bd004d3fe3ea34d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:13:00.613478 1995 consensus_queue.cc:1035] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c35421495e049d1b0b4487f06cc63ab" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:13:00.625929 2007 mvcc.cc:205] Tried to move back new op lower bound from 7334508874177859584 to 7334508874035118080. Current Snapshot: MvccSnapshot[applied={T|T < 7334508874177859584}]
I20260929 03:13:00.632295 1482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:13:00.632515 1482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:13:00.635273 1482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1cb3d743c38483ca25a22b30c5ea349: Sending DeleteTablet for 3 replicas of tablet 9f1e6258df274250a59b3bbdcbe510a6
I20260929 03:13:00.636205 1749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1e6258df274250a59b3bbdcbe510a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:00 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:39054
I20260929 03:13:00.636438 1615 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1e6258df274250a59b3bbdcbe510a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:00 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:41868
I20260929 03:13:00.636433 1865 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1e6258df274250a59b3bbdcbe510a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:00 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:37484
I20260929 03:13:00.637007 2014 tablet_replica.cc:330] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: stopping tablet replica
I20260929 03:13:00.637153 2015 tablet_replica.cc:330] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab: stopping tablet replica
I20260929 03:13:00.637480 2015 raft_consensus.cc:2229] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:13:00.637481 2014 raft_consensus.cc:2229] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:13:00.637718 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 1521
I20260929 03:13:00.637851 2015 raft_consensus.cc:2258] T 9f1e6258df274250a59b3bbdcbe510a6 P 0c35421495e049d1b0b4487f06cc63ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:00.638224 2016 tablet_replica.cc:330] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: stopping tablet replica
I20260929 03:13:00.638563 2016 raft_consensus.cc:2229] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:13:00.638880 2016 raft_consensus.cc:2258] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:00.640223 2014 raft_consensus.cc:2258] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:00.641839 2016 ts_tablet_manager.cc:1905] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:13:00.642151 2014 ts_tablet_manager.cc:1905] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:13:00.644918 2016 ts_tablet_manager.cc:1918] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:13:00.645109 2016 log.cc:1196] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/9f1e6258df274250a59b3bbdcbe510a6
I20260929 03:13:00.645517 2016 ts_tablet_manager.cc:1939] T 9f1e6258df274250a59b3bbdcbe510a6 P 45907cb081ae4faf8bd004d3fe3ea34d: Deleting consensus metadata
I20260929 03:13:00.646337 2014 ts_tablet_manager.cc:1918] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:13:00.646565 2014 log.cc:1196] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/9f1e6258df274250a59b3bbdcbe510a6
I20260929 03:13:00.646953 1468 catalog_manager.cc:4841] TS 45907cb081ae4faf8bd004d3fe3ea34d (127.20.174.67:37039): tablet 9f1e6258df274250a59b3bbdcbe510a6 (table test-table [id=991975ef1e354479b8c00f80a0204076]) successfully deleted
I20260929 03:13:00.647032 2014 ts_tablet_manager.cc:1939] T 9f1e6258df274250a59b3bbdcbe510a6 P 142b709315ac44ec932db284df70dacc: Deleting consensus metadata
W20260929 03:13:00.647416 1467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:43601 (error 108)
I20260929 03:13:00.648483 1469 catalog_manager.cc:4841] TS 142b709315ac44ec932db284df70dacc (127.20.174.66:42801): tablet 9f1e6258df274250a59b3bbdcbe510a6 (table test-table [id=991975ef1e354479b8c00f80a0204076]) successfully deleted
I20260929 03:13:00.648533 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:00.648227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:43601 (local address 127.0.0.1:42204)
0929 03:13:00.648375 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:00.648445 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:43601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 03:13:00.648656 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 1684
W20260929 03:13:00.650080 1467 catalog_manager.cc:4584] TS 0c35421495e049d1b0b4487f06cc63ab (127.20.174.65:43601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f1e6258df274250a59b3bbdcbe510a6: Network error: Client connection negotiation failed: client connection to 127.20.174.65:43601: connect: Connection refused (error 111)
I20260929 03:13:00.654865 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 1818
I20260929 03:13:00.660769 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 1450
I20260929 03:13:00.667493 21177 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:13:01.013621 21177 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:13:01.013739 21177 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:13:01.359609 21177 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:13:01.359 UTC [517] LOG: received smart shutdown request
2026-09-29 03:13:01.364 UTC [517] LOG: background worker "logical replication launcher" (PID 526) exited with exit code 1
2026-09-29 03:13:01.364 UTC [521] LOG: shutting down
2026-09-29 03:13:01.569 UTC [517] LOG: database system is shut down
2026-09-29T03:13:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53046 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:13:01 dist-test-slave-k1dg krb5kdc[2026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:13:01 dist-test-slave-k1dg krb5kdc[2026](info): set up 2 sockets
Sep 29 03:13:01 dist-test-slave-k1dg krb5kdc[2026](info): commencing operation
krb5kdc: starting...
W20260929 03:13:02.754715 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:13:02 dist-test-slave-k1dg krb5kdc[2026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:13:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:13:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:13:02.890756 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:39243
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:39243,127.20.174.125:33331,127.20.174.124:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:03.009965 2039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:03.010300 2039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:03.010380 2039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:03.014920 2039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:13:03.015023 2039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:03.015059 2039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:03.015097 2039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:13:03.015132 2039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:13:03.020984 2039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:39243,127.20.174.125:33331,127.20.174.124:38867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:39243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:03.022188 2039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:03.023401 2039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:13:03.032819 2039 server_base.cc:1034] running on GCE node
W20260929 03:13:03.032845 2047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:03.032742 2045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:03.032748 2044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:03.033907 2039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:03.035148 2039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:03.036332 2039 hybrid_clock.cc:648] HybridClock initialized: now 1790651583036300 us; error 54 us; skew 500 ppm
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:03.039390 2039 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:13:03.040359 2039 webserver.cc:460] Webserver started at http://127.20.174.126:39513/ using document root <none> and password file <none>
I20260929 03:13:03.040920 2039 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:03.040982 2039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:03.041150 2039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:03.043337 2039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "4b392ee7336b413f84dd0be0d9e11735"
format_stamp: "Formatted at 2026-09-29 03:13:03 on dist-test-slave-k1dg"
server_key: "e86c68ce480a8216af9e6ce02e98b586"
server_key_iv: "832dc54676734a29a1b839fc567fb776"
server_key_version: "encryptionkey@0"
I20260929 03:13:03.043896 2039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "4b392ee7336b413f84dd0be0d9e11735"
format_stamp: "Formatted at 2026-09-29 03:13:03 on dist-test-slave-k1dg"
server_key: "e86c68ce480a8216af9e6ce02e98b586"
server_key_iv: "832dc54676734a29a1b839fc567fb776"
server_key_version: "encryptionkey@0"
I20260929 03:13:03.047871 2039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:13:03.050609 2054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:03.051936 2039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:03.052091 2039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "4b392ee7336b413f84dd0be0d9e11735"
format_stamp: "Formatted at 2026-09-29 03:13:03 on dist-test-slave-k1dg"
server_key: "e86c68ce480a8216af9e6ce02e98b586"
server_key_iv: "832dc54676734a29a1b839fc567fb776"
server_key_version: "encryptionkey@0"
I20260929 03:13:03.052220 2039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:03.075080 2039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:03.078662 2039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:03.078881 2039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:03.086838 2039 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:39243
I20260929 03:13:03.086896 2106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:39243 every 8 connection(s)
I20260929 03:13:03.087885 2039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:13:03.090977 2107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:03.095074 2107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:13:03.097815 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 2039
I20260929 03:13:03.097960 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:13:03.098261 21177 external_mini_cluster.cc:1428] Setting key c24642e46220a83c85b446ca04b29fac
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.174.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.125@KRBTEST.COM" created.
I20260929 03:13:03.111502 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.097376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39243 (local address 127.0.0.1:57786)
0929 03:13:03.098001 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.098067 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:03.099228 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:03.099958 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:03.099964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:03.100546 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:03.100822 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.100837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.102017 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.102020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.102863 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.102874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.103558 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.103561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.103663 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.104293 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:03.104311 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:03.105807 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:03.107888 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.107898 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.107909 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.108240 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.108566 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.108570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.108572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.108692 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.109610 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:03.109617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:03.109768 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 03:13:03.109973 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 03:13:03.110409 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":130,"thread_start_us":199,"threads_started":1}
I20260929 03:13:03.111502 2110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.098474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.20.174.126:39243)
0929 03:13:03.098787 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.098800 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.098847 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:13:03.099555 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.099559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.099755 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.099964 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.100990 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.101883 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.103001 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.103398 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.103598 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.105987 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.105993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.106004 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.106024 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.107756 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.108340 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.108346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.108347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.108469 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.108983 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.108989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.108993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.109487 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.109633 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.110032 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.110360 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":208,"thread_start_us":104,"threads_started":1}
I20260929 03:13:03.114580 2107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:13:03.115422 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.115143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:33331 (local address 127.0.0.1:33742)
0929 03:13:03.115274 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.115354 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 03:13:03.115655 2056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.174.125:33331: connect: Connection refused (error 111)
I20260929 03:13:03.116549 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.116308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:33331 (local address 127.0.0.1:33752)
0929 03:13:03.116427 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.116482 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:13:03.116789 2107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.125:33331: Network error: Client connection negotiation failed: client connection to 127.20.174.125:33331: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab.
Entry for principal kudu/127.20.174.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab.
I20260929 03:13:03.126385 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.125:33331
--webserver_interface=127.20.174.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab
--principal=kudu/127.20.174.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:39243,127.20.174.125:33331,127.20.174.124:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:13:03.166108 2107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } attempt: 1
I20260929 03:13:03.167066 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.166700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:33331 (local address 127.0.0.1:33756)
0929 03:13:03.166888 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.166969 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 03:13:03.167876 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.167617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:33331 (local address 127.0.0.1:33760)
0929 03:13:03.167746 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.167809 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 03:13:03.168045 2107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.125:33331: Network error: Client connection negotiation failed: client connection to 127.20.174.125:33331: connect: Connection refused (error 111)
I20260929 03:13:03.236279 2107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } attempt: 2
I20260929 03:13:03.237257 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.236886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:33331 (local address 127.0.0.1:33774)
0929 03:13:03.237075 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.237154 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 03:13:03.238119 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.237870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:33331 (local address 127.0.0.1:33790)
0929 03:13:03.237989 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.238047 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 03:13:03.238408 2107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.125:33331: Network error: Client connection negotiation failed: client connection to 127.20.174.125:33331: connect: Connection refused (error 111)
W20260929 03:13:03.244773 2114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:03.245115 2114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:03.245204 2114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:03.249598 2114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:13:03.249708 2114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:03.249748 2114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:03.249784 2114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:13:03.249815 2114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:13:03.255726 2114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:39243,127.20.174.125:33331,127.20.174.124:38867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab
--principal=kudu/127.20.174.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.125:33331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.174.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:03.256995 2114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:03.258116 2114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:03.267349 2122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:03.267297 2119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:03.267311 2120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:03.267833 2114 server_base.cc:1034] running on GCE node
I20260929 03:13:03.268520 2114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:03.269829 2114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:03.271066 2114 hybrid_clock.cc:648] HybridClock initialized: now 1790651583271051 us; error 40 us; skew 500 ppm
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:03.274843 2114 init.cc:377] Logged in from keytab as kudu/127.20.174.125@KRBTEST.COM (short username kudu)
I20260929 03:13:03.275908 2114 webserver.cc:460] Webserver started at http://127.20.174.125:34549/ using document root <none> and password file <none>
I20260929 03:13:03.276528 2114 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:03.276628 2114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:03.276898 2114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:03.279165 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/data/instance:
uuid: "9b13c9c7a8774f5b9f9f5abad35afe35"
format_stamp: "Formatted at 2026-09-29 03:13:03 on dist-test-slave-k1dg"
server_key: "d1def45842a98bec1c8f16f534c02d21"
server_key_iv: "56a1a6f1cd2ad79000459808924b115d"
server_key_version: "encryptionkey@0"
I20260929 03:13:03.279796 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/wal/instance:
uuid: "9b13c9c7a8774f5b9f9f5abad35afe35"
format_stamp: "Formatted at 2026-09-29 03:13:03 on dist-test-slave-k1dg"
server_key: "d1def45842a98bec1c8f16f534c02d21"
server_key_iv: "56a1a6f1cd2ad79000459808924b115d"
server_key_version: "encryptionkey@0"
I20260929 03:13:03.283949 2114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:13:03.286671 2129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:03.288443 2114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:03.288578 2114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/wal
uuid: "9b13c9c7a8774f5b9f9f5abad35afe35"
format_stamp: "Formatted at 2026-09-29 03:13:03 on dist-test-slave-k1dg"
server_key: "d1def45842a98bec1c8f16f534c02d21"
server_key_iv: "56a1a6f1cd2ad79000459808924b115d"
server_key_version: "encryptionkey@0"
I20260929 03:13:03.288743 2114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:03.313100 2114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:03.316742 2114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:03.316999 2114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:03.324617 2114 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.125:33331
I20260929 03:13:03.324949 2181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.125:33331 every 8 connection(s)
I20260929 03:13:03.325702 2114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
I20260929 03:13:03.328944 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:03.329640 2107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } attempt: 3
I20260929 03:13:03.333820 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 2114
I20260929 03:13:03.333959 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-1/wal/instance
I20260929 03:13:03.334322 21177 external_mini_cluster.cc:1428] Setting key fbf4de726883a1c636a53cdf1eea070b
I20260929 03:13:03.334308 2182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.174.124@KRBTEST.COM; defaulting to no policy
I20260929 03:13:03.344995 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.330219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:33331 (local address 127.0.0.1:33804)
0929 03:13:03.330507 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.330521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:03.330606 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:03.333124 (+ 2518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:03.333127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:03.333344 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:03.333484 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.333492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.335051 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.335054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.335615 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.335625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.337298 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.337302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.337518 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.338351 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:03.338374 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:03.340246 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:03.342913 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.342918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.342922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.343184 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.343541 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.343545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.343546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.343640 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.344442 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:03.344448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:03.344568 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 03:13:03.344653 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:13:03.344799 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 03:13:03.346817 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.331044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.20.174.125:33331)
0929 03:13:03.331564 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.331579 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.331619 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:13:03.332436 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.332457 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.332812 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.333179 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.333665 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.334910 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.335778 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.336837 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.338151 (+ 1314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.340463 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.340472 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.340487 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.340520 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.342747 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.343307 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.343312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.343317 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.343429 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.343735 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.343740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.343742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.344309 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.345277 (+ 968us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.345431 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.345792 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.124@KRBTEST.COM" created.
I20260929 03:13:03.349195 2110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.336537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.20.174.126:39243)
0929 03:13:03.336683 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.336688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.337130 (+ 442us) server_negotiation.cc:407] Connection header received
0929 03:13:03.337624 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.337630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.337700 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.337807 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.338817 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.340271 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.341861 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.342234 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.342351 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.345253 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.345260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.345264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.345296 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.347056 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.347681 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.347686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.347689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.347791 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.348108 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.348115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.348119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.348446 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.348563 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.348921 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.349040 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:13:03.349208 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.336515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39243 (local address 127.0.0.1:57790)
0929 03:13:03.336993 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.337047 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:03.337397 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:03.337887 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:03.337898 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:03.338529 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:03.338680 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.338686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.340419 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.340422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.341636 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.341652 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.342389 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.342395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.342561 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.343155 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:03.343171 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:03.344928 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:03.347222 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.347230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.347236 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.347527 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.347897 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.347900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.347902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.347994 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.348563 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:03.348571 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:03.348690 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 03:13:03.348840 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 03:13:03.349041 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":188,"threads_started":1}
I20260929 03:13:03.349478 2107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:13:03.350539 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.350140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:49898)
0929 03:13:03.350347 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.350409 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 03:13:03.350662 2182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:13:03.351361 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.351068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:49912)
0929 03:13:03.351224 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.351291 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260929 03:13:03.351506 2107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:38867: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
I20260929 03:13:03.360198 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.351227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.20.174.125:33331)
0929 03:13:03.351365 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.351372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.351388 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:13:03.351530 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.351534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.351570 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.351636 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.352135 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.352648 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.353950 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.354192 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.354275 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.356855 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.356860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.356867 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.356887 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.358402 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.358838 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.358851 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.358853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.358929 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.359284 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.359298 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.359303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.359507 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.359612 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.359870 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.360046 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":32}
I20260929 03:13:03.360198 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.351109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:33331 (local address 127.0.0.1:33820)
0929 03:13:03.351336 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.351348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:03.351428 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:03.351662 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:03.351668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:03.351864 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:03.352013 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.352023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.352772 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.352778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.353739 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.353755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.354329 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.354335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.354508 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.355074 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:03.355094 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:03.356711 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:03.358542 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.358546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.358548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.358738 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.359026 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.359029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.359031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.359159 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.359612 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:03.359615 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:03.359702 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 03:13:03.359807 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:13:03.359979 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260929 03:13:03.361141 2182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:13:03.362036 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.361739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:49918)
0929 03:13:03.361876 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.361972 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 03:13:03.362747 2132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
I20260929 03:13:03.363684 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.363476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:49930)
0929 03:13:03.363576 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.363630 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:13:03.363821 2182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:38867: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab.
Entry for principal kudu/127.20.174.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab.
I20260929 03:13:03.366147 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.124:38867
--webserver_interface=127.20.174.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab
--principal=kudu/127.20.174.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:42043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:39243,127.20.174.125:33331,127.20.174.124:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:13:03.382678 2107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } attempt: 1
I20260929 03:13:03.383706 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.383291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:49936)
0929 03:13:03.383519 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.383599 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 03:13:03.384507 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.384233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:49950)
0929 03:13:03.384371 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.384433 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 03:13:03.384654 2107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:38867: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
I20260929 03:13:03.413126 2182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } attempt: 1
I20260929 03:13:03.414101 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.413800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:49960)
0929 03:13:03.413958 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.414022 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 03:13:03.414999 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.414733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:49974)
0929 03:13:03.414868 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.414930 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 03:13:03.415146 2182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:38867: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
I20260929 03:13:03.459863 2107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } attempt: 2
I20260929 03:13:03.460780 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.460420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:49980)
0929 03:13:03.460605 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.460679 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 03:13:03.461607 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.461355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:49988)
0929 03:13:03.461478 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.461536 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 03:13:03.461814 2107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:38867: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
I20260929 03:13:03.483333 2182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } attempt: 2
I20260929 03:13:03.484365 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.483999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:50000)
0929 03:13:03.484183 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.484258 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 03:13:03.485159 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.484919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:50010)
0929 03:13:03.485040 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.485096 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 03:13:03.485327 2182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:38867: Network error: Client connection negotiation failed: client connection to 127.20.174.124:38867: connect: Connection refused (error 111)
W20260929 03:13:03.489425 2189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:03.489722 2189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:03.489768 2189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:03.494082 2189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:13:03.494156 2189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:03.494176 2189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:03.494194 2189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:13:03.494210 2189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:13:03.499953 2189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:42043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:39243,127.20.174.125:33331,127.20.174.124:38867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab
--principal=kudu/127.20.174.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.124:38867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.174.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:03.501057 2189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:03.502072 2189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:03.511762 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:03.511730 2195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:03.511754 2189 server_base.cc:1034] running on GCE node
W20260929 03:13:03.511788 2194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:03.512859 2189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:03.514200 2189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:03.515439 2189 hybrid_clock.cc:648] HybridClock initialized: now 1790651583515425 us; error 39 us; skew 500 ppm
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:03.518682 2189 init.cc:377] Logged in from keytab as kudu/127.20.174.124@KRBTEST.COM (short username kudu)
I20260929 03:13:03.519652 2189 webserver.cc:460] Webserver started at http://127.20.174.124:41345/ using document root <none> and password file <none>
I20260929 03:13:03.520206 2189 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:03.520267 2189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:03.520460 2189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:03.522789 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/data/instance:
uuid: "dcf9dddefd8845d0b160431b4d116155"
format_stamp: "Formatted at 2026-09-29 03:13:03 on dist-test-slave-k1dg"
server_key: "b03bc21af260984860be9061d60ea217"
server_key_iv: "c9dd3bf63141c87bcac9d420c3f19cfb"
server_key_version: "encryptionkey@0"
I20260929 03:13:03.523366 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/wal/instance:
uuid: "dcf9dddefd8845d0b160431b4d116155"
format_stamp: "Formatted at 2026-09-29 03:13:03 on dist-test-slave-k1dg"
server_key: "b03bc21af260984860be9061d60ea217"
server_key_iv: "c9dd3bf63141c87bcac9d420c3f19cfb"
server_key_version: "encryptionkey@0"
I20260929 03:13:03.527516 2189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:13:03.530251 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:03.531598 2189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:03.531754 2189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/wal
uuid: "dcf9dddefd8845d0b160431b4d116155"
format_stamp: "Formatted at 2026-09-29 03:13:03 on dist-test-slave-k1dg"
server_key: "b03bc21af260984860be9061d60ea217"
server_key_iv: "c9dd3bf63141c87bcac9d420c3f19cfb"
server_key_version: "encryptionkey@0"
I20260929 03:13:03.531888 2189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:03.549401 2189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:03.553077 2189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:03.553308 2189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:03.561035 2107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } attempt: 3
I20260929 03:13:03.561201 2189 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.124:38867
I20260929 03:13:03.561517 2256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.124:38867 every 8 connection(s)
I20260929 03:13:03.562384 2189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
I20260929 03:13:03.563114 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 2189
I20260929 03:13:03.563263 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651413504048-21177-0/minicluster-data/master-2/wal/instance
I20260929 03:13:03.563584 21177 external_mini_cluster.cc:1428] Setting key 9a11e830d84ab2624a94ba4bfc24883d
I20260929 03:13:03.566157 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
I20260929 03:13:03.571666 2258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:13:03.575196 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.561566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:50024)
0929 03:13:03.561784 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.561807 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:03.561904 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:03.564656 (+ 2752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:03.564659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:03.564875 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:03.564992 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.564999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.566512 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.566516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.567317 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.567332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.568287 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.568291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.568386 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.568944 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:03.568964 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:03.570342 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:03.572974 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.572980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.572985 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.573253 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.573670 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.573676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.573681 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.573781 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.574595 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:03.574598 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:03.574708 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 03:13:03.574793 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:13:03.575022 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":56}
I20260929 03:13:03.576330 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.562398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.20.174.124:38867)
0929 03:13:03.562895 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.562921 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.562985 (+ 64us) server_negotiation.cc:407] Connection header received
0929 03:13:03.563807 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.563827 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.564295 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.564719 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.565172 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.566294 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.567463 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.568149 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.568324 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.570585 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.570592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.570610 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.570643 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.572837 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.573367 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.573373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.573379 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.573549 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.573914 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.573917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.573918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.574478 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.574935 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.575094 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.575410 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":360,"spinlock_wait_cycles":896,"thread_start_us":158,"threads_started":1}
I20260929 03:13:03.576337 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.564227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.20.174.124:38867)
0929 03:13:03.564496 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.564505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.564889 (+ 384us) server_negotiation.cc:407] Connection header received
0929 03:13:03.564928 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.564931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.564970 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.565043 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.566017 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.566741 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.567635 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.568158 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.568816 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.571379 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.571382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.571385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.571410 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.573182 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.573682 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.573686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.573688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.573782 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.574087 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.574093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.574098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.574469 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.575399 (+ 930us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.575473 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.575598 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":209,"spinlock_wait_cycles":2432,"thread_start_us":74,"threads_started":1}
I20260929 03:13:03.576489 2182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } attempt: 3
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:03.581465 2107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735: Bootstrap starting.
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
I20260929 03:13:03.584652 2107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:03.585304 2110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.573862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.20.174.126:39243)
0929 03:13:03.574015 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.574019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.574496 (+ 477us) server_negotiation.cc:407] Connection header received
0929 03:13:03.574669 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.574672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.574716 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.574797 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.575608 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.576376 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.577823 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.578161 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.578258 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.581627 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.581631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.581633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.581658 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.583270 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.583812 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.583815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.583817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.583926 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.584234 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.584237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.584239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.584466 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.584548 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.585033 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.585149 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":56,"spinlock_wait_cycles":5504}
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:03.585668 2107 log.cc:824] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:03.585826 2262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.573822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:39243 (local address 127.0.0.1:57794)
0929 03:13:03.574346 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.574400 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:03.574586 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:03.574853 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:03.574860 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:03.575328 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:03.575472 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.575480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.576798 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.576801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.577689 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.577698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.578412 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.578416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.578593 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.579796 (+ 1203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:03.579810 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:03.581471 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:03.583392 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.583401 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.583407 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.583695 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.584039 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.584042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.584044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.584131 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.584636 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:03.584642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:03.584784 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 03:13:03.585437 (+ 653us) client_negotiation.cc:240] Negotiation successful
0929 03:13:03.585637 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":44,"thread_start_us":175,"threads_started":1}
I20260929 03:13:03.587410 2258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:13:03.588812 2107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735: No bootstrap required, opened a new log
I20260929 03:13:03.589025 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.576998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:50048)
0929 03:13:03.577126 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.577140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:03.577220 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:03.577922 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:03.577926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:03.578116 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:03.578229 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.578236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.579668 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.579672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.580542 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.580555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.581270 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.581274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.581381 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.582118 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:03.582133 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:03.584192 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:03.587156 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.587159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.587161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.587476 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.587904 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.587907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.587909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.588021 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.588496 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:03.588499 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:03.588612 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 03:13:03.588730 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:13:03.588872 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260929 03:13:03.589471 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.577560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.20.174.124:38867)
0929 03:13:03.577682 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.577686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.577699 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:13:03.577730 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.577732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.577771 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.578372 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.578502 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.579571 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.580684 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.581003 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.581092 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.584375 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.584382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.584386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.584416 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.587017 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.587698 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.587702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.587704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.587791 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.588143 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.588148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.588149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.588353 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.588443 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.589226 (+ 783us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.589336 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":54}
I20260929 03:13:03.590977 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.579242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.20.174.126:39243)
0929 03:13:03.579833 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.579845 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.579858 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:13:03.579892 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.579894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.579934 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.580873 (+ 939us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.580966 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.581715 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.582940 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.583307 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.583783 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.586288 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.586292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.586295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.586322 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.588566 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.589078 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.589081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.589083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.589294 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.589608 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.589610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.589612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.589818 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.590570 (+ 752us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.590648 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.590767 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"spinlock_wait_cycles":3328,"thread_start_us":109,"threads_started":1}
I20260929 03:13:03.593070 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35: Bootstrap starting.
I20260929 03:13:03.593494 2107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } }
I20260929 03:13:03.593729 2107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:03.593777 2107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b392ee7336b413f84dd0be0d9e11735, State: Initialized, Role: FOLLOWER
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
I20260929 03:13:03.594296 2107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } }
I20260929 03:13:03.595806 2107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:13:03.596622 2182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:03.597922 2182 log.cc:824] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:03.598315 2262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.587930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:33331 (local address 127.0.0.1:33828)
0929 03:13:03.588047 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.588061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:03.588128 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:03.588597 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:03.588603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:03.588783 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:03.588885 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.588892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.590251 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.590255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.591168 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.591178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.591834 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.591841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.591935 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.592947 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:03.592961 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:03.594576 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:03.596550 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.596554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.596560 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.596818 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.597118 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.597124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.597130 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.597218 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.597691 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:03.597694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:03.597798 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 03:13:03.597927 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 03:13:03.598082 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 03:13:03.599289 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.588182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.20.174.125:33331)
0929 03:13:03.588288 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.588294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.588318 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:13:03.588355 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.588358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.588403 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.588479 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.589020 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.589752 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.591303 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.591609 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.591704 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.594724 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.594728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.594731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.594755 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.596417 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.596924 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.596927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.596929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.597011 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.597320 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.597323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.597326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.597557 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.597647 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.598639 (+ 992us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.598757 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31,"spinlock_wait_cycles":4864}
I20260929 03:13:03.599265 2266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } } }
I20260929 03:13:03.599694 2266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
I20260929 03:13:03.600147 2258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:13:03.602134 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35: No bootstrap required, opened a new log
W20260929 03:13:03.603793 2277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:13:03.603992 2277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:13:03.604827 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.592657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.20.174.125:33331)
0929 03:13:03.592982 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.592991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.593006 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:13:03.593042 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.593045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.593090 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.594182 (+ 1092us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.594559 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.595391 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.596635 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.597036 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.597485 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.600393 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.600397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.600400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.600427 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.602422 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.603002 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.603008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.603011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.603115 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.603438 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.603442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.603443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.603666 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.603866 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.604491 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.604612 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
I20260929 03:13:03.602032 2039 master_runner.cc:186] Error getting master registration for 127.20.174.125:33331: OK, instance_id { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" instance_seqno: 1790651583322384 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:13:03.606630 2039 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:13:03.607328 2182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } }
I20260929 03:13:03.607690 2182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:03.607764 2182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b13c9c7a8774f5b9f9f5abad35afe35, State: Initialized, Role: FOLLOWER
I20260929 03:13:03.608222 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:13:03.608294 2182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } }
I20260929 03:13:03.609838 2182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 03:13:03 dist-test-slave-k1dg krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651583, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
I20260929 03:13:03.615175 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.600984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.20.174.124:38867)
0929 03:13:03.606782 (+ 5798us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.606787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.606931 (+ 144us) server_negotiation.cc:407] Connection header received
0929 03:13:03.607061 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.607064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.607100 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.607203 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.607583 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.608258 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.609088 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.609319 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.609427 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.611631 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.611635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.611638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.611662 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.613310 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.613795 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.613798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.613800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.613877 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.614129 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.614132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.614134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.614329 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.614398 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.614843 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.614996 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5713}
I20260929 03:13:03.615665 2262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.600636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:38867 (local address 127.0.0.1:50060)
0929 03:13:03.606843 (+ 6207us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:03.606856 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:03.606977 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:03.607216 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:03.607219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:03.607396 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:03.607484 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.607492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.608392 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.608399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.608966 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:03.608975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.609426 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.609429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:03.609504 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.610014 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:03.610032 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:03.611494 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:03.613444 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.613448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.613450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.613694 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.613969 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:03.613972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:03.613974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:03.614048 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:03.614522 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:03.614524 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:03.614635 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 03:13:03.615375 (+ 740us) client_negotiation.cc:240] Negotiation successful
0929 03:13:03.615532 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6162}
W20260929 03:13:03.617488 2296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:13:03.617592 2296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:13:03.619185 2258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155: Bootstrap starting.
I20260929 03:13:03.619504 2280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } } }
I20260929 03:13:03.619668 2280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:13:03.622174 2258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:03.623725 2258 log.cc:824] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:03.626149 2258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155: No bootstrap required, opened a new log
I20260929 03:13:03.623932 2114 master_runner.cc:186] Error getting master registration for 127.20.174.124:38867: OK, instance_id { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" instance_seqno: 1790651583559190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:13:03.627749 2114 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:13:03.630573 2258 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } }
I20260929 03:13:03.630851 2258 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:03.630909 2258 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf9dddefd8845d0b160431b4d116155, State: Initialized, Role: FOLLOWER
I20260929 03:13:03.631443 2258 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } }
I20260929 03:13:03.632925 2258 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:13:03.633363 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.610571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.20.174.125:33331)
0929 03:13:03.610706 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.610712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.619530 (+ 8818us) server_negotiation.cc:407] Connection header received
0929 03:13:03.619911 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.619914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.619969 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.620057 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.620840 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.621397 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.622222 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.622584 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.622675 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.629252 (+ 6577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.629256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.629258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.629284 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.631261 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.631807 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.631812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.631813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.631902 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.632256 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.632260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.632264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.632510 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.632634 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.633023 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.633189 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:13:03.633440 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.619057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.20.174.124:38867)
0929 03:13:03.619148 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.619154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.623017 (+ 3863us) server_negotiation.cc:407] Connection header received
0929 03:13:03.623521 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.623524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.623569 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.623618 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.624795 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.625527 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.626814 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.627120 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.627199 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.629151 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.629158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.629161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.629194 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.631428 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.632067 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.632074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.632079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.632159 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.632452 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.632455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.632457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.632666 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.632798 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.633189 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.633298 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:13:03.642640 2189 master_runner.cc:263] No leader master found from master dcf9dddefd8845d0b160431b4d116155
I20260929 03:13:03.642732 2189 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 03:13:03.643577 2312 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:13:03.643666 2312 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:13:03.644857 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:03.618923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.20.174.126:39243)
0929 03:13:03.619054 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:03.619059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:03.634417 (+ 15358us) server_negotiation.cc:407] Connection header received
0929 03:13:03.634694 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:03.634698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:03.634754 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:03.634809 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:03.635709 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.636519 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.638040 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:03.638451 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:03.638538 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:03.640450 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:03.640455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:03.640458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:03.640485 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:03.642738 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.643353 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.643357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.643359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.643451 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:03.643773 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:03.643777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:03.643778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:03.644018 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:03.644125 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:03.644528 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 03:13:03.644679 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:13:03.650635 2301 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } } }
I20260929 03:13:03.650780 2301 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:13:03.805191 2210 tablet.cc:2369] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:13:03.818683 2135 tablet.cc:2369] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:13:03.831048 2060 tablet.cc:2369] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:13:03.856366 2301 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:13:03.856623 2301 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } }
I20260929 03:13:03.857651 2301 leader_election.cc:288] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b392ee7336b413f84dd0be0d9e11735 (127.20.174.126:39243), 9b13c9c7a8774f5b9f9f5abad35afe35 (127.20.174.125:33331)
I20260929 03:13:03.858290 2156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcf9dddefd8845d0b160431b4d116155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" is_pre_election: true
I20260929 03:13:03.858315 2079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcf9dddefd8845d0b160431b4d116155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b392ee7336b413f84dd0be0d9e11735" is_pre_election: true
I20260929 03:13:03.858660 2156 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcf9dddefd8845d0b160431b4d116155 in term 0.
I20260929 03:13:03.858736 2079 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcf9dddefd8845d0b160431b4d116155 in term 0.
I20260929 03:13:03.859139 2206 leader_election.cc:302] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b392ee7336b413f84dd0be0d9e11735, 9b13c9c7a8774f5b9f9f5abad35afe35, dcf9dddefd8845d0b160431b4d116155; no voters:
I20260929 03:13:03.859524 2301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:13:03.859615 2301 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:13:03.859675 2301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:03.860922 2301 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } }
I20260929 03:13:03.861327 2301 leader_election.cc:288] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [CANDIDATE]: Term 1 election: Requested vote from peers 4b392ee7336b413f84dd0be0d9e11735 (127.20.174.126:39243), 9b13c9c7a8774f5b9f9f5abad35afe35 (127.20.174.125:33331)
I20260929 03:13:03.861744 2079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcf9dddefd8845d0b160431b4d116155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b392ee7336b413f84dd0be0d9e11735"
I20260929 03:13:03.861745 2156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcf9dddefd8845d0b160431b4d116155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35"
I20260929 03:13:03.861907 2079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:03.862006 2156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:03.863508 2079 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcf9dddefd8845d0b160431b4d116155 in term 1.
I20260929 03:13:03.863538 2156 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcf9dddefd8845d0b160431b4d116155 in term 1.
I20260929 03:13:03.863880 2205 leader_election.cc:302] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b392ee7336b413f84dd0be0d9e11735, dcf9dddefd8845d0b160431b4d116155; no voters:
I20260929 03:13:03.864041 2301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:13:03.864332 2301 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [term 1 LEADER]: Becoming Leader. State: Replica: dcf9dddefd8845d0b160431b4d116155, State: Running, Role: LEADER
I20260929 03:13:03.864708 2301 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } }
I20260929 03:13:03.866772 2314 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcf9dddefd8845d0b160431b4d116155. Latest consensus state: current_term: 1 leader_uuid: "dcf9dddefd8845d0b160431b4d116155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } } }
I20260929 03:13:03.866918 2314 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:03.867317 2316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:13:03.868271 2316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:13:03.873544 2156 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [term 1 FOLLOWER]: Refusing update from remote peer dcf9dddefd8845d0b160431b4d116155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:03.873595 2079 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [term 1 FOLLOWER]: Refusing update from remote peer dcf9dddefd8845d0b160431b4d116155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:03.874258 2314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:13:03.874504 2301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:13:03.878624 2280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcf9dddefd8845d0b160431b4d116155. Latest consensus state: current_term: 1 leader_uuid: "dcf9dddefd8845d0b160431b4d116155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } } }
I20260929 03:13:03.878624 2266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcf9dddefd8845d0b160431b4d116155. Latest consensus state: current_term: 1 leader_uuid: "dcf9dddefd8845d0b160431b4d116155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } } }
I20260929 03:13:03.878758 2280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:13:03.878758 2266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:13:03.880496 2314 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcf9dddefd8845d0b160431b4d116155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } } }
I20260929 03:13:03.880622 2314 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:03.880525 2301 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcf9dddefd8845d0b160431b4d116155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } } }
I20260929 03:13:03.880702 2301 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:03.881140 2266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcf9dddefd8845d0b160431b4d116155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } } }
I20260929 03:13:03.881248 2266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:13:03.881392 2280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcf9dddefd8845d0b160431b4d116155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b392ee7336b413f84dd0be0d9e11735" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 39243 } } peers { permanent_uuid: "9b13c9c7a8774f5b9f9f5abad35afe35" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 33331 } } peers { permanent_uuid: "dcf9dddefd8845d0b160431b4d116155" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 38867 } } }
I20260929 03:13:03.881505 2280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:13:03.882086 2316 catalog_manager.cc:1347] Generated new cluster ID: 095245f0ff17463497cc294883948a37
I20260929 03:13:03.882211 2316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:13:03.883495 2319 mvcc.cc:205] Tried to move back new op lower bound from 7334508887538610176 to 7334508887512821760. Current Snapshot: MvccSnapshot[applied={T|T < 7334508887538610176}]
I20260929 03:13:03.904814 2316 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:13:03.906158 2316 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:13:03.917229 2316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcf9dddefd8845d0b160431b4d116155: Generated new TSK 0
I20260929 03:13:03.918243 2316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:13:04.605381 2277 catalog_manager.cc:1259] Loaded cluster ID: 095245f0ff17463497cc294883948a37
I20260929 03:13:04.605470 2277 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735: loading cluster ID for follower catalog manager: success
I20260929 03:13:04.607295 2277 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735: acquiring CA information for follower catalog manager: success
I20260929 03:13:04.608628 2277 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b392ee7336b413f84dd0be0d9e11735: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:13:04.619012 2296 catalog_manager.cc:1259] Loaded cluster ID: 095245f0ff17463497cc294883948a37
I20260929 03:13:04.619100 2296 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35: loading cluster ID for follower catalog manager: success
I20260929 03:13:04.620752 2296 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35: acquiring CA information for follower catalog manager: success
I20260929 03:13:04.621958 2296 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b13c9c7a8774f5b9f9f5abad35afe35: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:13:04.988771 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 2039
I20260929 03:13:04.995499 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 2114
I20260929 03:13:05.001572 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 2189
2026-09-29T03:13:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3312 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:13:05 dist-test-slave-k1dg krb5kdc[2331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:13:05 dist-test-slave-k1dg krb5kdc[2331](info): set up 2 sockets
Sep 29 03:13:05 dist-test-slave-k1dg krb5kdc[2331](info): commencing operation
krb5kdc: starting...
W20260929 03:13:06.056097 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:13:06 dist-test-slave-k1dg krb5kdc[2331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:13:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:13:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:13:06.203512 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44297
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:06.321267 2344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:06.321606 2344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:06.321650 2344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:06.326048 2344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:13:06.326131 2344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:06.326151 2344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:06.326170 2344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:13:06.326185 2344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:13:06.331791 2344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:44297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:06.333112 2344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:06.334219 2344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:06.343529 2350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:06.343583 2349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:06.343530 2352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:06.343977 2344 server_base.cc:1034] running on GCE node
I20260929 03:13:06.344568 2344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:06.345804 2344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:06.346983 2344 hybrid_clock.cc:648] HybridClock initialized: now 1790651586346964 us; error 34 us; skew 500 ppm
Sep 29 03:13:06 dist-test-slave-k1dg krb5kdc[2331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651586, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:06.350019 2344 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:13:06.351051 2344 webserver.cc:460] Webserver started at http://127.20.174.126:38767/ using document root <none> and password file <none>
I20260929 03:13:06.351667 2344 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:06.351735 2344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:06.351979 2344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:06.354185 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "e957a52089214a9b92921a2e080c2161"
format_stamp: "Formatted at 2026-09-29 03:13:06 on dist-test-slave-k1dg"
server_key: "54bcf48fcab6cbf1d0c87bd009f576e9"
server_key_iv: "669c1a706a8ab2c1dc054e34eb31336b"
server_key_version: "encryptionkey@0"
I20260929 03:13:06.354882 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "e957a52089214a9b92921a2e080c2161"
format_stamp: "Formatted at 2026-09-29 03:13:06 on dist-test-slave-k1dg"
server_key: "54bcf48fcab6cbf1d0c87bd009f576e9"
server_key_iv: "669c1a706a8ab2c1dc054e34eb31336b"
server_key_version: "encryptionkey@0"
I20260929 03:13:06.359242 2344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:13:06.361964 2359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:06.363418 2344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:06.363593 2344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "e957a52089214a9b92921a2e080c2161"
format_stamp: "Formatted at 2026-09-29 03:13:06 on dist-test-slave-k1dg"
server_key: "54bcf48fcab6cbf1d0c87bd009f576e9"
server_key_iv: "669c1a706a8ab2c1dc054e34eb31336b"
server_key_version: "encryptionkey@0"
I20260929 03:13:06.363768 2344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:06.377076 2344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:06.380286 2344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:06.380517 2344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:06.388374 2344 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:44297
I20260929 03:13:06.388396 2411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:44297 every 8 connection(s)
I20260929 03:13:06.389467 2344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:13:06.390908 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 2344
I20260929 03:13:06.391063 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:13:06.391440 21177 external_mini_cluster.cc:1428] Setting key 7e96dea5e09ce1dbfae251fa23df5cc3
I20260929 03:13:06.393352 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:06.401038 2412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161: Bootstrap starting.
Sep 29 03:13:06 dist-test-slave-k1dg krb5kdc[2331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:06.403970 2412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:06.404800 2412 log.cc:824] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:06.407105 2412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161: No bootstrap required, opened a new log
I20260929 03:13:06.408397 2415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:06.392810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.20.174.126:44297)
0929 03:13:06.393255 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:06.393280 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:06.393317 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:13:06.394055 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:06.394074 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:06.394457 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:06.394808 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:06.395902 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:06.397017 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:06.398346 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:06.399086 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:06.399235 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:06.402719 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:06.402725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:06.402745 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:06.402787 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:06.404873 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:06.405468 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:06.405474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:06.405479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:06.405605 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:06.405972 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:06.405978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:06.405980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:06.406505 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:06.406740 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:06.407086 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 03:13:06.407558 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":153,"threads_started":1}
I20260929 03:13:06.410920 2412 raft_consensus.cc:348] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e957a52089214a9b92921a2e080c2161" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44297 } }
I20260929 03:13:06.411218 2412 raft_consensus.cc:374] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:06.411288 2412 raft_consensus.cc:729] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e957a52089214a9b92921a2e080c2161, State: Initialized, Role: FOLLOWER
I20260929 03:13:06.411868 2412 consensus_queue.cc:260] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e957a52089214a9b92921a2e080c2161" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44297 } }
I20260929 03:13:06.412051 2412 raft_consensus.cc:388] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:13:06.412086 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:13:06.412132 2412 raft_consensus.cc:482] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:13:06.412235 2412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:06.413546 2412 raft_consensus.cc:504] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e957a52089214a9b92921a2e080c2161" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44297 } }
I20260929 03:13:06.413955 2412 leader_election.cc:302] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e957a52089214a9b92921a2e080c2161; no voters:
I20260929 03:13:06.414330 2412 leader_election.cc:288] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:13:06.414541 2417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:13:06.414852 2417 raft_consensus.cc:686] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [term 1 LEADER]: Becoming Leader. State: Replica: e957a52089214a9b92921a2e080c2161, State: Running, Role: LEADER
I20260929 03:13:06.415201 2417 consensus_queue.cc:237] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e957a52089214a9b92921a2e080c2161" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44297 } }
I20260929 03:13:06.415807 2412 sys_catalog.cc:564] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:13:06.417124 2419 sys_catalog.cc:455] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e957a52089214a9b92921a2e080c2161. Latest consensus state: current_term: 1 leader_uuid: "e957a52089214a9b92921a2e080c2161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e957a52089214a9b92921a2e080c2161" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44297 } } }
I20260929 03:13:06.417279 2419 sys_catalog.cc:458] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:06.417634 2418 sys_catalog.cc:455] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e957a52089214a9b92921a2e080c2161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e957a52089214a9b92921a2e080c2161" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44297 } } }
I20260929 03:13:06.417742 2418 sys_catalog.cc:458] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:06.417766 2426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:13:06.421402 2426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:13:06.427809 2426 catalog_manager.cc:1347] Generated new cluster ID: ee133f8082424b7186122ffe6894581e
I20260929 03:13:06.427907 2426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:13:06.435966 2426 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:13:06.436934 2426 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:13:06.450101 2426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e957a52089214a9b92921a2e080c2161: Generated new TSK 0
I20260929 03:13:06.450925 2426 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:13:06.469165 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 2344
2026-09-29T03:13:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1465 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:13:06 dist-test-slave-k1dg krb5kdc[2439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:13:06 dist-test-slave-k1dg krb5kdc[2439](info): set up 2 sockets
Sep 29 03:13:06 dist-test-slave-k1dg krb5kdc[2439](info): commencing operation
krb5kdc: starting...
W20260929 03:13:07.551412 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:13:07 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:13:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:13:07.694621 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:45109
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:45109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:07.814000 2452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:07.814384 2452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:07.814527 2452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:07.818846 2452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:13:07.818980 2452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:07.819016 2452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:07.819059 2452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:13:07.819094 2452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:13:07.824895 2452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:45109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:45109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:07.826150 2452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:07.827342 2452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:07.836585 2460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:07.836612 2458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:07.836789 2452 server_base.cc:1034] running on GCE node
W20260929 03:13:07.836609 2457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:07.837527 2452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:07.838837 2452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:07.840045 2452 hybrid_clock.cc:648] HybridClock initialized: now 1790651587840010 us; error 58 us; skew 500 ppm
Sep 29 03:13:07 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651587, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:07.853067 2452 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:13:07.854115 2452 webserver.cc:460] Webserver started at http://127.20.174.126:41371/ using document root <none> and password file <none>
I20260929 03:13:07.854808 2452 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:07.854908 2452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:07.855154 2452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:07.857287 2452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "44219ceaaceb4843a5c3449b4a40a166"
format_stamp: "Formatted at 2026-09-29 03:13:07 on dist-test-slave-k1dg"
server_key: "b19e4851130c38b8e137344cd6991bbb"
server_key_iv: "80a739e3a27b86571ee032eb88c86deb"
server_key_version: "encryptionkey@0"
I20260929 03:13:07.857877 2452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "44219ceaaceb4843a5c3449b4a40a166"
format_stamp: "Formatted at 2026-09-29 03:13:07 on dist-test-slave-k1dg"
server_key: "b19e4851130c38b8e137344cd6991bbb"
server_key_iv: "80a739e3a27b86571ee032eb88c86deb"
server_key_version: "encryptionkey@0"
I20260929 03:13:07.862030 2452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260929 03:13:07.864851 2467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:07.866602 2452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:07.866780 2452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "44219ceaaceb4843a5c3449b4a40a166"
format_stamp: "Formatted at 2026-09-29 03:13:07 on dist-test-slave-k1dg"
server_key: "b19e4851130c38b8e137344cd6991bbb"
server_key_iv: "80a739e3a27b86571ee032eb88c86deb"
server_key_version: "encryptionkey@0"
I20260929 03:13:07.866915 2452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:07.887224 2452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:07.890890 2452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:07.891106 2452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:07.899163 2452 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:45109
I20260929 03:13:07.899485 2519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:45109 every 8 connection(s)
I20260929 03:13:07.900245 2452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:13:07.902132 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 2452
I20260929 03:13:07.902262 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:13:07.902668 21177 external_mini_cluster.cc:1428] Setting key 9bb4627b39261292cb1d1e66fcb33191
I20260929 03:13:07.903615 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:07.910969 2520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Bootstrap starting.
Sep 29 03:13:07 dist-test-slave-k1dg krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:07.913913 2520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:07.914862 2520 log.cc:824] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:07.916868 2520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: No bootstrap required, opened a new log
I20260929 03:13:07.918762 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:07.904211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.20.174.126:45109)
0929 03:13:07.904675 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:07.904690 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:07.904728 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:13:07.905484 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:07.905501 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:07.905848 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:07.906134 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:07.907147 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:07.908479 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:07.909813 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:07.910310 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:07.910520 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:07.913470 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:07.913475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:07.913490 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:07.913520 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:07.915568 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:07.916117 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:07.916123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:07.916128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:07.916247 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:07.916564 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:07.916574 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:07.916576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:07.917070 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:07.917278 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:07.917580 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 03:13:07.917918 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":152,"threads_started":1}
I20260929 03:13:07.921191 2520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:07.921510 2520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:07.921572 2520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44219ceaaceb4843a5c3449b4a40a166, State: Initialized, Role: FOLLOWER
I20260929 03:13:07.922017 2520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:07.922151 2520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:13:07.922223 2520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:13:07.922344 2520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:07.923395 2520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:07.923751 2520 leader_election.cc:302] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44219ceaaceb4843a5c3449b4a40a166; no voters:
I20260929 03:13:07.924093 2520 leader_election.cc:288] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:13:07.924242 2525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:13:07.924542 2525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 1 LEADER]: Becoming Leader. State: Replica: 44219ceaaceb4843a5c3449b4a40a166, State: Running, Role: LEADER
I20260929 03:13:07.924868 2525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:07.925386 2520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:13:07.926676 2527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44219ceaaceb4843a5c3449b4a40a166. Latest consensus state: current_term: 1 leader_uuid: "44219ceaaceb4843a5c3449b4a40a166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } } }
I20260929 03:13:07.926824 2527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:07.927233 2534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:13:07.927775 2526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44219ceaaceb4843a5c3449b4a40a166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } } }
I20260929 03:13:07.927932 2526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:07.930212 2534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:13:07.935873 2534 catalog_manager.cc:1347] Generated new cluster ID: c4eeee4770fd408b95d5bc09355c7dc0
I20260929 03:13:07.935952 2534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:13:07.961305 2534 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:13:07.962666 2534 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:13:07.973023 2534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Generated new TSK 0
I20260929 03:13:07.973968 2534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:13:08.011221 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:45109
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:08.130537 2546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:08.130915 2546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:08.130980 2546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:08.138599 2546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:08.138666 2546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:08.138777 2546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:13:08.144533 2546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:08.145736 2546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:08.146970 2546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:08.157392 2554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:08.157490 2552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:08.157483 2546 server_base.cc:1034] running on GCE node
W20260929 03:13:08.157507 2551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:08.158227 2546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:08.159004 2546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:08.160214 2546 hybrid_clock.cc:648] HybridClock initialized: now 1790651588160184 us; error 53 us; skew 500 ppm
Sep 29 03:13:08 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651588, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:08.163210 2546 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:13:08.164371 2546 webserver.cc:460] Webserver started at http://127.20.174.65:36937/ using document root <none> and password file <none>
I20260929 03:13:08.165113 2546 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:08.165207 2546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:08.165462 2546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:08.168388 2546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "3c0fa121c26942b0b1f012c711241262"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "d43db2b657743c4d12393f6e72551c05"
server_key_iv: "2136042d05f9d10a6f7c92d30267c2b7"
server_key_version: "encryptionkey@0"
I20260929 03:13:08.169090 2546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "3c0fa121c26942b0b1f012c711241262"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "d43db2b657743c4d12393f6e72551c05"
server_key_iv: "2136042d05f9d10a6f7c92d30267c2b7"
server_key_version: "encryptionkey@0"
I20260929 03:13:08.174075 2546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 03:13:08.176831 2561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:08.178238 2546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:13:08.178478 2546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "3c0fa121c26942b0b1f012c711241262"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "d43db2b657743c4d12393f6e72551c05"
server_key_iv: "2136042d05f9d10a6f7c92d30267c2b7"
server_key_version: "encryptionkey@0"
I20260929 03:13:08.178627 2546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:08.191347 2546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:08.194728 2546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:08.194933 2546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:08.195616 2546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:08.196485 2546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:13:08.196564 2546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:08.196635 2546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:13:08.196688 2546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:08.206169 2546 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:33967
I20260929 03:13:08.206190 2674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:33967 every 8 connection(s)
I20260929 03:13:08.207144 2546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:13:08.208315 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 2546
I20260929 03:13:08.208459 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:13:08.208752 21177 external_mini_cluster.cc:1428] Setting key fe17989c7d5e166738131544587f362f
Sep 29 03:13:08 dist-test-slave-k1dg krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651588, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:13:08.223573 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.209161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:33289 (local address 127.20.174.126:45109)
0929 03:13:08.209312 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:08.209319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:08.210444 (+ 1125us) server_negotiation.cc:407] Connection header received
0929 03:13:08.211910 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:08.211913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:08.211962 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:08.212035 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:08.213821 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.214383 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.216457 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.216691 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.216779 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.219732 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:08.219736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:08.219738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:08.219764 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:08.221300 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:08.221878 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:08.221881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:08.221883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:08.221970 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:08.222335 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:08.222340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:08.222342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:08.222610 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:08.222722 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:08.223246 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 03:13:08.223384 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:13:08.224433 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.209674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:33289)
0929 03:13:08.210204 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:08.210265 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:08.211647 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:08.212241 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:08.212252 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:08.212850 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:08.213628 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.213652 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.214635 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.214643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.216294 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.216310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.216878 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.216884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.217102 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.217775 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:08.217800 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:08.219529 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:08.221459 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:08.221465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:08.221482 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:08.221764 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:08.222081 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:08.222084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:08.222086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:08.222235 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:08.222722 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:08.222730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:08.223002 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 03:13:08.223207 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 03:13:08.223557 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":123,"threads_started":1}
I20260929 03:13:08.225823 2675 heartbeater.cc:347] Connected to a master server at 127.20.174.126:45109
I20260929 03:13:08.226208 2675 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:08.226902 2675 heartbeater.cc:511] Master 127.20.174.126:45109 requested a full tablet report, sending...
I20260929 03:13:08.228883 2484 ts_manager.cc:194] Registered new tserver with Master: 3c0fa121c26942b0b1f012c711241262 (127.20.174.65:33967)
I20260929 03:13:08.230687 2484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:33289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:13:08.242064 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:45109
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:08.356709 2680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:08.356997 2680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:08.357041 2680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:08.361143 2680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:08.361208 2680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:08.361271 2680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:13:08.366734 2680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:08.367841 2680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:08.368949 2680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:08.379276 2686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:08.379293 2685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:08.379271 2688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:08.379475 2680 server_base.cc:1034] running on GCE node
I20260929 03:13:08.379993 2680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:08.380669 2680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:08.381876 2680 hybrid_clock.cc:648] HybridClock initialized: now 1790651588381854 us; error 34 us; skew 500 ppm
Sep 29 03:13:08 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651588, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:08.384964 2680 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:13:08.385838 2680 webserver.cc:460] Webserver started at http://127.20.174.66:36831/ using document root <none> and password file <none>
I20260929 03:13:08.386493 2680 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:08.386576 2680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:08.386757 2680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:08.388950 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "53297af6141d43cda47548443c3dc00d"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "11cee5a9f76d7b280723e5f1394d893a"
server_key_iv: "14ede0069e2b5fe8f4030b4a0b96f977"
server_key_version: "encryptionkey@0"
I20260929 03:13:08.389483 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "53297af6141d43cda47548443c3dc00d"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "11cee5a9f76d7b280723e5f1394d893a"
server_key_iv: "14ede0069e2b5fe8f4030b4a0b96f977"
server_key_version: "encryptionkey@0"
I20260929 03:13:08.393471 2680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:13:08.396165 2695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:08.397569 2680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 03:13:08.397676 2680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "53297af6141d43cda47548443c3dc00d"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "11cee5a9f76d7b280723e5f1394d893a"
server_key_iv: "14ede0069e2b5fe8f4030b4a0b96f977"
server_key_version: "encryptionkey@0"
I20260929 03:13:08.397774 2680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:08.415078 2680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:08.418946 2680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:08.419199 2680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:08.419921 2680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:08.420853 2680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:13:08.420929 2680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:08.421007 2680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:13:08.421059 2680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:08.430631 2680 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:38263
I20260929 03:13:08.430717 2808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:38263 every 8 connection(s)
I20260929 03:13:08.431649 2680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:13:08.439180 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 2680
I20260929 03:13:08.439337 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:13:08.439628 21177 external_mini_cluster.cc:1428] Setting key 3be4cf83dd4751022d09cfdb1367a310
Sep 29 03:13:08 dist-test-slave-k1dg krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651588, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:08.448387 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.433863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:50005 (local address 127.20.174.126:45109)
0929 03:13:08.434025 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:08.434028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:08.434973 (+ 945us) server_negotiation.cc:407] Connection header received
0929 03:13:08.436471 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:08.436475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:08.436533 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:08.436621 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:08.437946 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.438559 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.439898 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.440141 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.440263 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.443352 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:08.443358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:08.443362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:08.443398 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:08.445267 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:08.446162 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:08.446165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:08.446166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:08.446264 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:08.446874 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:08.446877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:08.446878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:08.447099 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:08.447200 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:08.448095 (+ 895us) server_negotiation.cc:299] Negotiation successful
0929 03:13:08.448242 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:13:08.449751 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.434218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:50005)
0929 03:13:08.434781 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:08.434849 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:08.436192 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:08.436831 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:08.436838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:08.437214 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:08.437788 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.437803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.438698 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.438702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.439742 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.439756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.440320 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.440325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.440514 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.441185 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:08.441204 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:08.443096 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:08.445425 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:08.445435 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:08.445459 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:08.446000 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:08.446392 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:08.446395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:08.446398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:08.446669 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:08.447259 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:08.447270 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:08.447723 (+ 453us) client_negotiation.cc:769] Sending connection context
0929 03:13:08.447994 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 03:13:08.448451 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":188,"threads_started":1}
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:13:08.451560 2809 heartbeater.cc:347] Connected to a master server at 127.20.174.126:45109
I20260929 03:13:08.451954 2809 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:08.452626 2809 heartbeater.cc:511] Master 127.20.174.126:45109 requested a full tablet report, sending...
I20260929 03:13:08.453943 2484 ts_manager.cc:194] Registered new tserver with Master: 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:13:08.454747 2484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:50005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:13:08.475605 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:45109
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:08.592331 2814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:08.592610 2814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:08.592654 2814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:08.596835 2814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:08.596909 2814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:08.596972 2814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:13:08.602615 2814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:08.603680 2814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:08.604724 2814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:08.614960 2819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:08.614992 2814 server_base.cc:1034] running on GCE node
W20260929 03:13:08.614997 2822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:08.615005 2820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:08.615723 2814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:08.616395 2814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:08.617606 2814 hybrid_clock.cc:648] HybridClock initialized: now 1790651588617539 us; error 86 us; skew 500 ppm
Sep 29 03:13:08 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651588, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:08.620687 2814 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:13:08.621670 2814 webserver.cc:460] Webserver started at http://127.20.174.67:38757/ using document root <none> and password file <none>
I20260929 03:13:08.622272 2814 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:08.622366 2814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:08.622634 2814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:08.624769 2814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "3ea91a0fb644441796e207f1e9d8d86d"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "17d4134ebfb91bceaeac76327ec40e2e"
server_key_iv: "5e8f41697a80004abd6061bc84bf58cd"
server_key_version: "encryptionkey@0"
I20260929 03:13:08.625423 2814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "3ea91a0fb644441796e207f1e9d8d86d"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "17d4134ebfb91bceaeac76327ec40e2e"
server_key_iv: "5e8f41697a80004abd6061bc84bf58cd"
server_key_version: "encryptionkey@0"
I20260929 03:13:08.629823 2814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:13:08.632572 2829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:08.634011 2814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:13:08.634159 2814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "3ea91a0fb644441796e207f1e9d8d86d"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "17d4134ebfb91bceaeac76327ec40e2e"
server_key_iv: "5e8f41697a80004abd6061bc84bf58cd"
server_key_version: "encryptionkey@0"
I20260929 03:13:08.634284 2814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:08.656152 2814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:08.659473 2814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:08.659690 2814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:08.660326 2814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:08.661211 2814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:13:08.661286 2814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:08.661352 2814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:13:08.661396 2814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:08.670867 2814 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:34581
I20260929 03:13:08.670951 2942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:34581 every 8 connection(s)
I20260929 03:13:08.671972 2814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:13:08.672953 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 2814
I20260929 03:13:08.673120 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:13:08.673430 21177 external_mini_cluster.cc:1428] Setting key 3dfe3964959331e484865c1854ee2404
Sep 29 03:13:08 dist-test-slave-k1dg krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651588, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:08.687214 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.674307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:48445 (local address 127.20.174.126:45109)
0929 03:13:08.674480 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:08.674484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:08.675525 (+ 1041us) server_negotiation.cc:407] Connection header received
0929 03:13:08.677055 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:08.677058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:08.677115 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:08.677189 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:08.678503 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.679045 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.680244 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.680478 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.680576 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.683491 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:08.683494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:08.683496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:08.683523 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:08.684874 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:08.685424 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:08.685427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:08.685428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:08.685513 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:08.685887 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:08.685889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:08.685890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:08.686099 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:08.686251 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:08.686919 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 03:13:08.687053 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:13:08.688253 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.674823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:48445)
0929 03:13:08.675316 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:08.675370 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:08.676831 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:08.677330 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:08.677337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:08.677796 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:08.678303 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.678318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.679162 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.679165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.680099 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.680109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.680624 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.680626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.680768 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.681368 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:08.681385 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:08.683313 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:08.685000 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:08.685005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:08.685018 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:08.685303 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:08.685612 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:08.685615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:08.685616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:08.685786 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:08.686281 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:08.686287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:08.686624 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 03:13:08.686878 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 03:13:08.687237 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
I20260929 03:13:08.689405 2943 heartbeater.cc:347] Connected to a master server at 127.20.174.126:45109
I20260929 03:13:08.689626 2943 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:08.690150 2943 heartbeater.cc:511] Master 127.20.174.126:45109 requested a full tablet report, sending...
I20260929 03:13:08.691262 2484 ts_manager.cc:194] Registered new tserver with Master: 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581)
I20260929 03:13:08.691846 2484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:48445
I20260929 03:13:08.700073 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:13:08.709625 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.702292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.20.174.126:45109)
0929 03:13:08.702504 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:08.702514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:08.702595 (+ 81us) server_negotiation.cc:407] Connection header received
0929 03:13:08.702686 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:08.702688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:08.702729 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:08.702799 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:08.703476 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.704012 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.704852 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.705063 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.705136 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.706493 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:08.706496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:08.706499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:08.706521 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:08.707883 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:08.708389 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:08.708392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:08.708394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:08.708498 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:08.708771 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:08.708773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:08.708774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:08.709003 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:08.709111 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:08.709388 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 03:13:08.709498 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 03:13:08.712951 2484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:13:08.716216 2484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:13:08.732062 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.725656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:34581 (local address 127.0.0.1:41158)
0929 03:13:08.726515 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:08.726529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:08.726655 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:08.727713 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:08.727716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:08.727730 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:08.727840 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.727848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.729275 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.729286 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.730668 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.730675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.731495 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.731499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.731618 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.731656 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:13:08.731771 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 03:13:08.731902 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":175,"thread_start_us":77,"threads_started":1}
I20260929 03:13:08.732074 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.726625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.20.174.67:34581)
0929 03:13:08.727138 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:08.727153 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:08.727177 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:13:08.727255 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:08.727262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:08.727521 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:08.727703 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:08.727933 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.729097 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.730797 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.731363 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.731500 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.731662 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:08.731809 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:13:08.731910 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":91,"threads_started":1}
I20260929 03:13:08.732298 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.725669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.20.174.65:33967)
0929 03:13:08.726196 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:08.726204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:08.726583 (+ 379us) server_negotiation.cc:407] Connection header received
0929 03:13:08.726859 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:08.726869 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:08.727101 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:08.727282 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:08.727892 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.728844 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.730615 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.731460 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.731560 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.731695 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:08.732050 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 03:13:08.732154 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":135,"threads_started":1}
I20260929 03:13:08.732666 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.725522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:33967 (local address 127.0.0.1:33076)
0929 03:13:08.726406 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:08.726487 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:08.726655 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:08.727507 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:08.727520 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:08.727542 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:08.727729 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.727736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.728964 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.728967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.730475 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.730486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.731757 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.731763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.731878 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.731902 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:13:08.732449 (+ 547us) client_negotiation.cc:240] Negotiation successful
0929 03:13:08.732528 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":59,"thread_start_us":98,"threads_started":1}
I20260929 03:13:08.732713 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.726388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:38263 (local address 127.0.0.1:50768)
0929 03:13:08.726713 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:08.726729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:08.726866 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:08.728155 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:08.728158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:08.728177 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:08.728266 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.728272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.729817 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.729820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.731037 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.731047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.732111 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.732114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.732474 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.732496 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:13:08.732549 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:13:08.732603 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260929 03:13:08.733425 2959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.727042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.20.174.66:38263)
0929 03:13:08.727625 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:08.727639 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:08.727685 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:13:08.727762 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:08.727767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:08.728000 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:08.728163 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:08.728372 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.729497 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.731189 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.731987 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.732969 (+ 982us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.733122 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:08.733170 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 03:13:08.733257 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":169,"threads_started":1}
I20260929 03:13:08.735858 2743 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae41204a9dc426a87e199435243670c (DEFAULT_TABLE table=test-table [id=f25f6ae8690243af89e9368ae33510d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:08.735720 2875 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae41204a9dc426a87e199435243670c (DEFAULT_TABLE table=test-table [id=f25f6ae8690243af89e9368ae33510d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:08.736099 2609 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae41204a9dc426a87e199435243670c (DEFAULT_TABLE table=test-table [id=f25f6ae8690243af89e9368ae33510d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:08.736896 2743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae41204a9dc426a87e199435243670c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:08.736919 2609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae41204a9dc426a87e199435243670c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:08.736914 2875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae41204a9dc426a87e199435243670c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:08.744170 2960 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Bootstrap starting.
I20260929 03:13:08.744952 2961 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Bootstrap starting.
I20260929 03:13:08.745187 2962 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: Bootstrap starting.
I20260929 03:13:08.746986 2962 tablet_bootstrap.cc:654] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:08.747064 2960 tablet_bootstrap.cc:654] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:08.747192 2961 tablet_bootstrap.cc:654] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:08.748050 2961 log.cc:824] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:08.748050 2960 log.cc:824] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:08.748112 2962 log.cc:824] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:08.750190 2961 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: No bootstrap required, opened a new log
I20260929 03:13:08.750269 2960 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: No bootstrap required, opened a new log
I20260929 03:13:08.750422 2961 ts_tablet_manager.cc:1397] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:13:08.750496 2960 ts_tablet_manager.cc:1397] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260929 03:13:08.750810 2962 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: No bootstrap required, opened a new log
I20260929 03:13:08.750980 2962 ts_tablet_manager.cc:1397] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:13:08.753949 2961 raft_consensus.cc:348] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:08.754274 2961 raft_consensus.cc:374] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:08.754333 2961 raft_consensus.cc:729] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0fa121c26942b0b1f012c711241262, State: Initialized, Role: FOLLOWER
I20260929 03:13:08.754480 2960 raft_consensus.cc:348] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:08.754742 2960 raft_consensus.cc:374] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:08.754806 2960 raft_consensus.cc:729] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53297af6141d43cda47548443c3dc00d, State: Initialized, Role: FOLLOWER
I20260929 03:13:08.754884 2961 consensus_queue.cc:260] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:08.754823 2962 raft_consensus.cc:348] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:08.755115 2962 raft_consensus.cc:374] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:08.755175 2962 raft_consensus.cc:729] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea91a0fb644441796e207f1e9d8d86d, State: Initialized, Role: FOLLOWER
I20260929 03:13:08.755255 2960 consensus_queue.cc:260] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:08.755728 2675 heartbeater.cc:503] Master 127.20.174.126:45109 was elected leader, sending a full tablet report...
I20260929 03:13:08.756125 2961 ts_tablet_manager.cc:1428] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260929 03:13:08.756232 2960 ts_tablet_manager.cc:1428] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:13:08.756537 2809 heartbeater.cc:503] Master 127.20.174.126:45109 was elected leader, sending a full tablet report...
I20260929 03:13:08.757040 2962 consensus_queue.cc:260] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:08.757658 2943 heartbeater.cc:503] Master 127.20.174.126:45109 was elected leader, sending a full tablet report...
I20260929 03:13:08.757980 2962 ts_tablet_manager.cc:1428] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260929 03:13:08.923204 2944 tablet.cc:2369] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:13:08.932960 2810 tablet.cc:2369] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:13:08.958786 2676 tablet.cc:2369] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:13:08.964061 2967 raft_consensus.cc:482] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:13:08.964246 2967 raft_consensus.cc:504] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:08.965453 2967 leader_election.cc:288] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c0fa121c26942b0b1f012c711241262 (127.20.174.65:33967), 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581)
I20260929 03:13:08.969347 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.965780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:53037 (local address 127.20.174.65:33967)
0929 03:13:08.965942 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:08.965947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:08.966031 (+ 84us) server_negotiation.cc:407] Connection header received
0929 03:13:08.966239 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:08.966246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:08.966289 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:08.966379 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:08.966676 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.967220 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.968358 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.968782 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.968926 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.968981 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:08.969149 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:13:08.969218 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 03:13:08.969359 2969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.965626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:33967 (local address 127.20.174.66:53037)
0929 03:13:08.965917 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:08.965934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:08.966118 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:08.966392 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:08.966402 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:08.966420 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:08.966565 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.966571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.967343 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.967348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.968220 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.968231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.968913 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.968917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.969012 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.969043 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:13:08.969135 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:13:08.969207 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
I20260929 03:13:08.969507 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.965939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:36921 (local address 127.20.174.67:34581)
0929 03:13:08.966050 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:08.966054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:08.966250 (+ 196us) server_negotiation.cc:407] Connection header received
0929 03:13:08.966409 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:08.966416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:08.966519 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:08.966599 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:08.966902 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.967403 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.968478 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.968887 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.969019 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.969073 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:08.969264 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 03:13:08.969335 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:13:08.969728 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:08.965745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:34581 (local address 127.20.174.66:36921)
0929 03:13:08.966149 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:08.966160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:08.966297 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:08.966624 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:08.966626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:08.966643 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:08.966765 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.966771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.967518 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.967523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.968341 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:08.968351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:08.969010 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:08.969014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:08.969121 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:08.969149 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 03:13:08.969234 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:13:08.969307 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":82,"threads_started":1}
I20260929 03:13:08.970125 2629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae41204a9dc426a87e199435243670c" candidate_uuid: "53297af6141d43cda47548443c3dc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0fa121c26942b0b1f012c711241262" is_pre_election: true
I20260929 03:13:08.970515 2629 raft_consensus.cc:2454] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53297af6141d43cda47548443c3dc00d in term 0.
I20260929 03:13:08.970533 2897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae41204a9dc426a87e199435243670c" candidate_uuid: "53297af6141d43cda47548443c3dc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ea91a0fb644441796e207f1e9d8d86d" is_pre_election: true
I20260929 03:13:08.970901 2897 raft_consensus.cc:2454] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53297af6141d43cda47548443c3dc00d in term 0.
I20260929 03:13:08.971041 2697 leader_election.cc:302] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0fa121c26942b0b1f012c711241262, 53297af6141d43cda47548443c3dc00d; no voters:
I20260929 03:13:08.971338 2967 raft_consensus.cc:2781] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:13:08.971429 2967 raft_consensus.cc:482] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:13:08.971483 2967 raft_consensus.cc:3037] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:08.972534 2967 raft_consensus.cc:504] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:08.972968 2967 leader_election.cc:288] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [CANDIDATE]: Term 1 election: Requested vote from peers 3c0fa121c26942b0b1f012c711241262 (127.20.174.65:33967), 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581)
I20260929 03:13:08.973367 2897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae41204a9dc426a87e199435243670c" candidate_uuid: "53297af6141d43cda47548443c3dc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ea91a0fb644441796e207f1e9d8d86d"
I20260929 03:13:08.973364 2629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae41204a9dc426a87e199435243670c" candidate_uuid: "53297af6141d43cda47548443c3dc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0fa121c26942b0b1f012c711241262"
I20260929 03:13:08.973502 2629 raft_consensus.cc:3037] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:08.973502 2897 raft_consensus.cc:3037] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:08.974885 2629 raft_consensus.cc:2454] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53297af6141d43cda47548443c3dc00d in term 1.
I20260929 03:13:08.974885 2897 raft_consensus.cc:2454] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53297af6141d43cda47548443c3dc00d in term 1.
I20260929 03:13:08.975256 2698 leader_election.cc:302] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ea91a0fb644441796e207f1e9d8d86d, 53297af6141d43cda47548443c3dc00d; no voters:
I20260929 03:13:08.975507 2967 raft_consensus.cc:2781] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:13:08.975795 2967 raft_consensus.cc:686] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 1 LEADER]: Becoming Leader. State: Replica: 53297af6141d43cda47548443c3dc00d, State: Running, Role: LEADER
I20260929 03:13:08.976192 2967 consensus_queue.cc:237] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:08.979861 2482 catalog_manager.cc:5495] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 53297af6141d43cda47548443c3dc00d (127.20.174.66). New cstate: current_term: 1 leader_uuid: "53297af6141d43cda47548443c3dc00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } health_report { overall_health: HEALTHY } } }
I20260929 03:13:09.028364 2959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.024322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.20.174.66:38263)
0929 03:13:09.024475 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:09.024480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:09.024495 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:13:09.024670 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:09.024673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:09.024731 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:09.024850 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:13:09.025126 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.025634 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.026705 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.026927 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.027040 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.027413 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:13:09.027931 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:13:09.028073 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:09.028150 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 03:13:09.028213 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:13:09.035945 2629 raft_consensus.cc:1261] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Refusing update from remote peer 53297af6141d43cda47548443c3dc00d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:09.035945 2897 raft_consensus.cc:1261] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d [term 1 FOLLOWER]: Refusing update from remote peer 53297af6141d43cda47548443c3dc00d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:09.036799 2967 consensus_queue.cc:1035] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:13:09.037308 2971 consensus_queue.cc:1035] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:13:09.045729 2975 mvcc.cc:205] Tried to move back new op lower bound from 7334508908681621504 to 7334508908449718272. Current Snapshot: MvccSnapshot[applied={T|T < 7334508908681621504}]
I20260929 03:13:09.047819 2976 mvcc.cc:205] Tried to move back new op lower bound from 7334508908681621504 to 7334508908449718272. Current Snapshot: MvccSnapshot[applied={T|T < 7334508908681621504}]
I20260929 03:13:09.046598 2974 mvcc.cc:205] Tried to move back new op lower bound from 7334508908681621504 to 7334508908449718272. Current Snapshot: MvccSnapshot[applied={T|T < 7334508908681621504}]
I20260929 03:13:09.056031 2482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:13:09.056196 2482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:13:09.058974 2482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Sending DeleteTablet for 3 replicas of tablet 6ae41204a9dc426a87e199435243670c
I20260929 03:13:09.059912 2743 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ae41204a9dc426a87e199435243670c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:09 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:50768
I20260929 03:13:09.059911 2875 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ae41204a9dc426a87e199435243670c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:09 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:41158
I20260929 03:13:09.059908 2609 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ae41204a9dc426a87e199435243670c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:09 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:33076
I20260929 03:13:09.060513 2983 tablet_replica.cc:330] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: stopping tablet replica
I20260929 03:13:09.060680 2985 tablet_replica.cc:330] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: stopping tablet replica
I20260929 03:13:09.060708 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 2546
I20260929 03:13:09.060918 2983 raft_consensus.cc:2229] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:13:09.061290 2983 raft_consensus.cc:2258] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:09.061717 2984 tablet_replica.cc:330] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: stopping tablet replica
I20260929 03:13:09.062062 2984 raft_consensus.cc:2229] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:13:09.062546 2984 raft_consensus.cc:2258] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:09.063704 2983 ts_tablet_manager.cc:1905] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:13:09.066911 2983 ts_tablet_manager.cc:1918] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:13:09.067031 2983 log.cc:1196] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/6ae41204a9dc426a87e199435243670c
I20260929 03:13:09.067371 2983 ts_tablet_manager.cc:1939] T 6ae41204a9dc426a87e199435243670c P 3ea91a0fb644441796e207f1e9d8d86d: Deleting consensus metadata
W20260929 03:13:09.068902 2469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:33967 (error 108)
I20260929 03:13:09.068912 2470 catalog_manager.cc:4841] TS 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581): tablet 6ae41204a9dc426a87e199435243670c (table test-table [id=f25f6ae8690243af89e9368ae33510d0]) successfully deleted
I20260929 03:13:09.069159 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 2680
I20260929 03:13:09.070405 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.069798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:33967 (local address 127.0.0.1:33090)
0929 03:13:09.070196 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:09.070282 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:33967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283}
W20260929 03:13:09.070588 2469 catalog_manager.cc:4584] TS 3c0fa121c26942b0b1f012c711241262 (127.20.174.65:33967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ae41204a9dc426a87e199435243670c: Network error: Client connection negotiation failed: client connection to 127.20.174.65:33967: connect: Connection refused (error 111)
I20260929 03:13:09.074534 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 2814
I20260929 03:13:09.075927 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.074811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:38263 (local address 127.0.0.1:50792)
0929 03:13:09.075777 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:09.075837 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.66:38263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":875}
I20260929 03:13:09.080271 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 2452
I20260929 03:13:09.087113 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:45109
--webserver_interface=127.20.174.126
--webserver_port=41371
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:45109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:09.203774 2987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:09.204133 2987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:09.204223 2987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:09.208564 2987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:13:09.208683 2987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:09.208722 2987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:09.208763 2987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:13:09.208799 2987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:13:09.214636 2987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:45109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:45109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=41371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:09.215852 2987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:09.216969 2987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:09.230798 2994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:09.230816 2993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:09.230816 2996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:09.231123 2987 server_base.cc:1034] running on GCE node
I20260929 03:13:09.231788 2987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:09.233114 2987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:09.234361 2987 hybrid_clock.cc:648] HybridClock initialized: now 1790651589234334 us; error 60 us; skew 500 ppm
Sep 29 03:13:09 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651589, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:09.237447 2987 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:13:09.238703 2987 webserver.cc:460] Webserver started at http://127.20.174.126:41371/ using document root <none> and password file <none>
I20260929 03:13:09.239403 2987 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:09.239516 2987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:09.243676 2987 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 03:13:09.246138 3003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:09.247637 2987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:13:09.247828 2987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "44219ceaaceb4843a5c3449b4a40a166"
format_stamp: "Formatted at 2026-09-29 03:13:07 on dist-test-slave-k1dg"
server_key: "b19e4851130c38b8e137344cd6991bbb"
server_key_iv: "80a739e3a27b86571ee032eb88c86deb"
server_key_version: "encryptionkey@0"
I20260929 03:13:09.248494 2987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:09.265324 2987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:09.268939 2987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:09.269191 2987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:09.277468 2987 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:45109
I20260929 03:13:09.277787 3055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:45109 every 8 connection(s)
I20260929 03:13:09.278820 2987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:13:09.283632 3056 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:13:09.284584 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 2987
I20260929 03:13:09.285087 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:33967
--local_ip_for_outbound_sockets=127.20.174.65
--tserver_master_addrs=127.20.174.126:45109
--webserver_port=36937
--webserver_interface=127.20.174.65
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:13:09.285956 3056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Bootstrap starting.
I20260929 03:13:09.304587 3056 log.cc:824] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:09.314623 3056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:13:09.315050 3056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Bootstrap complete.
I20260929 03:13:09.319314 3056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:09.319818 3056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44219ceaaceb4843a5c3449b4a40a166, State: Initialized, Role: FOLLOWER
I20260929 03:13:09.320384 3056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:09.320503 3056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:13:09.320595 3056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:13:09.320714 3056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:13:09.322805 3056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:09.323154 3056 leader_election.cc:302] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44219ceaaceb4843a5c3449b4a40a166; no voters:
I20260929 03:13:09.323501 3056 leader_election.cc:288] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:13:09.323588 3060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:13:09.323817 3060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 2 LEADER]: Becoming Leader. State: Replica: 44219ceaaceb4843a5c3449b4a40a166, State: Running, Role: LEADER
I20260929 03:13:09.324120 3060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:09.324815 3056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:13:09.325127 3060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44219ceaaceb4843a5c3449b4a40a166. Latest consensus state: current_term: 2 leader_uuid: "44219ceaaceb4843a5c3449b4a40a166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } } }
I20260929 03:13:09.325251 3060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:09.325629 3064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:13:09.325655 3061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "44219ceaaceb4843a5c3449b4a40a166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } } }
I20260929 03:13:09.325909 3061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:09.330945 3064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:13:09.331753 3064 catalog_manager.cc:1259] Loaded cluster ID: c4eeee4770fd408b95d5bc09355c7dc0
I20260929 03:13:09.331827 3064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:13:09.333313 3064 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 03:13:09.338136 3064 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 03:13:09.344426 3064 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:13:09.345321 3064 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Loaded TSK: 0
I20260929 03:13:09.346642 3064 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:13:09.406052 3058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:09.406423 3058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:09.406538 3058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:09.410881 3058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:09.410980 3058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:09.411078 3058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:13:09.416713 3058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:33967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=36937
--tserver_master_addrs=127.20.174.126:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:09.417852 3058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:09.419023 3058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:09.430239 3086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:09.430305 3083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:09.430239 3084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:09.430656 3058 server_base.cc:1034] running on GCE node
I20260929 03:13:09.431169 3058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:09.431963 3058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:09.433161 3058 hybrid_clock.cc:648] HybridClock initialized: now 1790651589433127 us; error 48 us; skew 500 ppm
Sep 29 03:13:09 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651589, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:09.436273 3058 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:13:09.437384 3058 webserver.cc:460] Webserver started at http://127.20.174.65:36937/ using document root <none> and password file <none>
I20260929 03:13:09.437985 3058 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:09.438114 3058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:09.442682 3058 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 03:13:09.445050 3093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:09.446669 3058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:09.446813 3058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "3c0fa121c26942b0b1f012c711241262"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "d43db2b657743c4d12393f6e72551c05"
server_key_iv: "2136042d05f9d10a6f7c92d30267c2b7"
server_key_version: "encryptionkey@0"
I20260929 03:13:09.447511 3058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:09.463426 3058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:09.467200 3058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:09.467447 3058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:09.468200 3058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:09.469493 3100 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:13:09.474146 3058 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:13:09.474231 3058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 03:13:09.474287 3058 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:13:09.476308 3058 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:13:09.476382 3058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 03:13:09.476655 3100 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Bootstrap starting.
I20260929 03:13:09.487902 3058 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:33967
I20260929 03:13:09.487954 3207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:33967 every 8 connection(s)
I20260929 03:13:09.489385 3058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:13:09.492738 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 3058
I20260929 03:13:09.493256 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:38263
--local_ip_for_outbound_sockets=127.20.174.66
--tserver_master_addrs=127.20.174.126:45109
--webserver_port=36831
--webserver_interface=127.20.174.66
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:13:09.499333 3100 log.cc:824] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:13:09 dist-test-slave-k1dg krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651589, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:09.508798 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.492021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:46121)
0929 03:13:09.492662 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:09.492730 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:09.494090 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:09.495781 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:09.495790 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:09.496228 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:09.496701 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.496715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.498117 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.498120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.499131 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.499142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.500088 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.500093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.500319 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.501031 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:09.501049 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:09.503080 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:09.505368 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:09.505375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:09.505389 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:09.505694 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:09.506097 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:09.506100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:09.506102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:09.506247 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:09.506940 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:09.506946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:09.507202 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 03:13:09.507401 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 03:13:09.507800 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":180,"threads_started":1}
I20260929 03:13:09.508783 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.492028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:46121 (local address 127.20.174.126:45109)
0929 03:13:09.494573 (+ 2545us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:09.494607 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:09.494661 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:13:09.494989 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:09.495001 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:09.495349 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:09.495714 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:09.496915 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.497923 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.499286 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.499918 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.500774 (+ 856us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.503323 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:09.503329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:09.503343 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:09.503380 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:09.505219 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:09.505845 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:09.505851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:09.505855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:09.505985 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:09.506349 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:09.506354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:09.506356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:09.506802 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:09.506960 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:09.507532 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 03:13:09.507829 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":150,"threads_started":1}
I20260929 03:13:09.510239 3100 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:13:09.510820 3100 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Bootstrap complete.
I20260929 03:13:09.511709 3208 heartbeater.cc:347] Connected to a master server at 127.20.174.126:45109
I20260929 03:13:09.511968 3208 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:09.512346 3100 ts_tablet_manager.cc:1397] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Time spent bootstrapping tablet: real 0.036s user 0.030s sys 0.004s
I20260929 03:13:09.512552 3208 heartbeater.cc:511] Master 127.20.174.126:45109 requested a full tablet report, sending...
I20260929 03:13:09.515259 3020 ts_manager.cc:194] Registered new tserver with Master: 3c0fa121c26942b0b1f012c711241262 (127.20.174.65:33967)
I20260929 03:13:09.515466 3100 raft_consensus.cc:348] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:09.516067 3100 raft_consensus.cc:729] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0fa121c26942b0b1f012c711241262, State: Initialized, Role: FOLLOWER
I20260929 03:13:09.516613 3100 consensus_queue.cc:260] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:09.517652 3100 ts_tablet_manager.cc:1428] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:13:09.519068 3020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:46121
I20260929 03:13:09.521831 3208 heartbeater.cc:503] Master 127.20.174.126:45109 was elected leader, sending a full tablet report...
I20260929 03:13:09.525457 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.518351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:33967 (local address 127.0.0.1:33102)
0929 03:13:09.520020 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:09.520067 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:09.520208 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:09.521347 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:09.521355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:09.521383 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:09.521516 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.521523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.522814 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.522820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.523886 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.523897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.524803 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.524807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.524932 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.524982 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 03:13:09.525161 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 03:13:09.525306 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1497,"thread_start_us":147,"threads_started":1}
I20260929 03:13:09.525457 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.518668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.20.174.65:33967)
0929 03:13:09.519060 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:09.519071 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:09.520176 (+ 1105us) server_negotiation.cc:407] Connection header received
0929 03:13:09.520368 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:09.520373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:09.521017 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:09.521181 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:09.521662 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.522699 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.524017 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.524663 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.524794 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.524934 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:09.525207 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 03:13:09.525318 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":109,"threads_started":1}
I20260929 03:13:09.526893 3142 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ae41204a9dc426a87e199435243670c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:09 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:33102
I20260929 03:13:09.527844 3217 tablet_replica.cc:330] stopping tablet replica
I20260929 03:13:09.528213 3217 raft_consensus.cc:2229] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:13:09.528479 3217 raft_consensus.cc:2258] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:09.529522 3217 ts_tablet_manager.cc:1905] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:13:09.533250 3217 ts_tablet_manager.cc:1918] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:13:09.533372 3217 log.cc:1196] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal/wals/6ae41204a9dc426a87e199435243670c
I20260929 03:13:09.533679 3217 ts_tablet_manager.cc:1939] T 6ae41204a9dc426a87e199435243670c P 3c0fa121c26942b0b1f012c711241262: Deleting consensus metadata
I20260929 03:13:09.535496 3005 catalog_manager.cc:4841] TS 3c0fa121c26942b0b1f012c711241262 (127.20.174.65:33967): tablet 6ae41204a9dc426a87e199435243670c (table test-table [id=f25f6ae8690243af89e9368ae33510d0]) successfully deleted
W20260929 03:13:09.620252 3212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:09.620635 3212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:09.620716 3212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:09.625358 3212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:09.625488 3212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:09.625587 3212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:13:09.631413 3212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:38263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=36831
--tserver_master_addrs=127.20.174.126:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:09.632517 3212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:09.633662 3212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:09.644810 3223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:09.644896 3225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:09.644810 3212 server_base.cc:1034] running on GCE node
W20260929 03:13:09.644891 3222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:09.645615 3212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:09.646417 3212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:09.647643 3212 hybrid_clock.cc:648] HybridClock initialized: now 1790651589647612 us; error 50 us; skew 500 ppm
Sep 29 03:13:09 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651589, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:09.651057 3212 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:13:09.652349 3212 webserver.cc:460] Webserver started at http://127.20.174.66:36831/ using document root <none> and password file <none>
I20260929 03:13:09.653131 3212 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:09.653244 3212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:09.657859 3212 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:13:09.660504 3232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:09.662171 3212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:13:09.662339 3212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "53297af6141d43cda47548443c3dc00d"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "11cee5a9f76d7b280723e5f1394d893a"
server_key_iv: "14ede0069e2b5fe8f4030b4a0b96f977"
server_key_version: "encryptionkey@0"
I20260929 03:13:09.663043 3212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:09.691385 3212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:09.695039 3212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:09.695266 3212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:09.695916 3212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:09.697062 3239 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:13:09.701700 3212 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:13:09.701892 3212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 03:13:09.702026 3212 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:13:09.704056 3212 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:13:09.704128 3212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 03:13:09.704416 3239 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Bootstrap starting.
I20260929 03:13:09.715979 3212 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:38263
I20260929 03:13:09.716013 3346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:38263 every 8 connection(s)
I20260929 03:13:09.717077 3212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:13:09.724480 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 3212
I20260929 03:13:09.724808 3239 log.cc:824] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:09.725082 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:34581
--local_ip_for_outbound_sockets=127.20.174.67
--tserver_master_addrs=127.20.174.126:45109
--webserver_port=38757
--webserver_interface=127.20.174.67
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:13:09 dist-test-slave-k1dg krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651589, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:09.733513 3239 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:13:09.733922 3239 tablet_bootstrap.cc:492] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Bootstrap complete.
I20260929 03:13:09.735257 3239 ts_tablet_manager.cc:1397] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20260929 03:13:09.735584 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.719720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:34809 (local address 127.20.174.126:45109)
0929 03:13:09.719897 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:09.719902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:09.720685 (+ 783us) server_negotiation.cc:407] Connection header received
0929 03:13:09.722005 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:09.722008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:09.722064 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:09.722159 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:09.723944 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.724783 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.726695 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.727128 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.727228 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.730766 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:09.730773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:09.730778 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:09.730819 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:09.732790 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:09.733484 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:09.733491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:09.733494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:09.733601 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:09.734132 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:09.734136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:09.734137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:09.734421 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:09.734555 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:09.735255 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 03:13:09.735411 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:13:09.736768 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.719916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:34809)
0929 03:13:09.720465 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:09.720534 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:09.721724 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:09.722463 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:09.722480 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:09.723015 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:09.723738 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.723760 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.725015 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.725018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.726505 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.726517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.727462 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.727466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.727728 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.728479 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:09.728505 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:09.730520 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:09.732948 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:09.732953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:09.732967 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:09.733345 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:09.733734 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:09.733740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:09.733744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:09.733955 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:09.734555 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:09.734564 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:09.734929 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 03:13:09.735169 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 03:13:09.735604 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260929 03:13:09.738037 3347 heartbeater.cc:347] Connected to a master server at 127.20.174.126:45109
I20260929 03:13:09.738289 3347 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:09.738090 3239 raft_consensus.cc:348] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:09.738688 3239 raft_consensus.cc:729] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53297af6141d43cda47548443c3dc00d, State: Initialized, Role: FOLLOWER
I20260929 03:13:09.738860 3347 heartbeater.cc:511] Master 127.20.174.126:45109 requested a full tablet report, sending...
I20260929 03:13:09.739198 3239 consensus_queue.cc:260] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:09.740244 3239 ts_tablet_manager.cc:1428] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 03:13:09.741170 3020 ts_manager.cc:194] Registered new tserver with Master: 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263)
I20260929 03:13:09.742553 3020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:34809
I20260929 03:13:09.744650 3347 heartbeater.cc:503] Master 127.20.174.126:45109 was elected leader, sending a full tablet report...
Sep 29 03:13:09 dist-test-slave-k1dg krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651589, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.66@KRBTEST.COM
I20260929 03:13:09.753772 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.742147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:38263 (local address 127.0.0.1:50796)
0929 03:13:09.742278 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:09.742291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:09.742364 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:09.743199 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:09.743202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:09.743670 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:09.743801 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.743809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.745256 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.745259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.745916 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.745926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.746796 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.746800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.746901 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.747544 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:09.747560 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:09.749327 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:09.751601 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:09.751614 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:09.751620 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:09.751955 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:09.752352 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:09.752356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:09.752357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:09.752489 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:09.753136 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:09.753143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:09.753316 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 03:13:09.753438 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 03:13:09.753600 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 03:13:09.754050 3353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.742353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.20.174.66:38263)
0929 03:13:09.742713 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:09.742729 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:09.742752 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:13:09.742817 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:09.742823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:09.743032 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:09.743204 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:09.743933 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.745145 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.746040 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.746608 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.746723 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.749499 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:09.749508 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:09.749525 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:09.749551 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:09.751480 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:09.752141 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:09.752146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:09.752147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:09.752245 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:09.752592 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:09.752595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:09.752597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:09.753013 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:09.753118 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:09.753558 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 03:13:09.753757 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":141,"threads_started":1}
I20260929 03:13:09.755380 3281 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ae41204a9dc426a87e199435243670c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:09 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:50796
I20260929 03:13:09.756351 3354 tablet_replica.cc:330] stopping tablet replica
I20260929 03:13:09.756685 3354 raft_consensus.cc:2229] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:13:09.756953 3354 raft_consensus.cc:2258] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:09.758116 3354 ts_tablet_manager.cc:1905] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:13:09.763098 3354 ts_tablet_manager.cc:1918] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:13:09.763221 3354 log.cc:1196] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/6ae41204a9dc426a87e199435243670c
I20260929 03:13:09.763502 3354 ts_tablet_manager.cc:1939] T 6ae41204a9dc426a87e199435243670c P 53297af6141d43cda47548443c3dc00d: Deleting consensus metadata
I20260929 03:13:09.765482 3007 catalog_manager.cc:4841] TS 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263): tablet 6ae41204a9dc426a87e199435243670c (table test-table [id=f25f6ae8690243af89e9368ae33510d0]) successfully deleted
W20260929 03:13:09.855018 3351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:09.855285 3351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:09.855330 3351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:09.859428 3351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:09.859495 3351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:09.859556 3351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:13:09.865002 3351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:34581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=38757
--tserver_master_addrs=127.20.174.126:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:09.866058 3351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:09.867265 3351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:09.877496 3359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:09.877496 3360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:09.877514 3362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:09.877854 3351 server_base.cc:1034] running on GCE node
I20260929 03:13:09.878419 3351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:09.879195 3351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:09.880399 3351 hybrid_clock.cc:648] HybridClock initialized: now 1790651589880366 us; error 50 us; skew 500 ppm
Sep 29 03:13:09 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651589, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:09.883490 3351 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:13:09.884510 3351 webserver.cc:460] Webserver started at http://127.20.174.67:38757/ using document root <none> and password file <none>
I20260929 03:13:09.885103 3351 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:09.885214 3351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:09.889371 3351 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:13:09.891738 3369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:09.893313 3351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:13:09.893534 3351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "3ea91a0fb644441796e207f1e9d8d86d"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "17d4134ebfb91bceaeac76327ec40e2e"
server_key_iv: "5e8f41697a80004abd6061bc84bf58cd"
server_key_version: "encryptionkey@0"
I20260929 03:13:09.894124 3351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:09.907706 3351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:09.911413 3351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:09.911638 3351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:09.912281 3351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:09.913152 3351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:13:09.913232 3351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:09.913305 3351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:13:09.913348 3351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:09.923121 3351 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:34581
I20260929 03:13:09.923136 3482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:34581 every 8 connection(s)
I20260929 03:13:09.924244 3351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:13:09.933809 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 3351
Sep 29 03:13:09 dist-test-slave-k1dg krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651589, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:09.940371 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.926399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:45149 (local address 127.20.174.126:45109)
0929 03:13:09.926551 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:09.926557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:09.927595 (+ 1038us) server_negotiation.cc:407] Connection header received
0929 03:13:09.929071 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:09.929074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:09.929125 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:09.929206 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:09.930614 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.931170 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.932444 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.932668 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.932830 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.935857 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:09.935861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:09.935863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:09.935889 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:09.937770 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:09.938346 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:09.938350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:09.938351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:09.938498 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:09.938926 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:09.938929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:09.938930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:09.939164 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:09.939266 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:09.940027 (+ 761us) server_negotiation.cc:299] Negotiation successful
0929 03:13:09.940159 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:13:09.941408 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.926932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:45149)
0929 03:13:09.927426 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:09.927472 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:09.928759 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:09.929402 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:09.929409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:09.929896 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:09.930454 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.930469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.931303 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.931306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.932296 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.932307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.932949 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.932952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.933100 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.933798 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:09.933820 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:09.935665 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:09.937896 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:09.937903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:09.937920 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:09.938229 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:09.938605 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:09.938609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:09.938613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:09.938811 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:09.939300 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:09.939306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:09.939647 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 03:13:09.939886 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 03:13:09.940193 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260929 03:13:09.942852 3483 heartbeater.cc:347] Connected to a master server at 127.20.174.126:45109
I20260929 03:13:09.943142 3483 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:09.943781 3483 heartbeater.cc:511] Master 127.20.174.126:45109 requested a full tablet report, sending...
I20260929 03:13:09.944509 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.934673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.20.174.126:45109)
0929 03:13:09.934938 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:09.934943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:09.935148 (+ 205us) server_negotiation.cc:407] Connection header received
0929 03:13:09.935825 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:09.935827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:09.935870 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:09.935971 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:09.936936 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.937685 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.938647 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.938996 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.939075 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.940940 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:09.940944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:09.940947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:09.940972 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:09.942770 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:09.943307 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:09.943310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:09.943312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:09.943402 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:09.943665 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:09.943669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:09.943670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:09.943898 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:09.944102 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:09.944210 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 03:13:09.944335 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260929 03:13:09.945030 3020 ts_manager.cc:194] Registered new tserver with Master: 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581)
I20260929 03:13:09.945725 3020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:45149
I20260929 03:13:09.946710 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:13:09.951267 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.947362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.20.174.126:45109)
0929 03:13:09.947505 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:09.947509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:09.947757 (+ 248us) server_negotiation.cc:407] Connection header received
0929 03:13:09.947892 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:09.947895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:09.947980 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:09.948062 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:13:09.948309 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.948806 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.949766 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.949992 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.950077 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.950385 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:13:09.950902 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:13:09.950995 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:09.951082 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 03:13:09.951132 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:13:09.951992 3019 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:13:09.955343 3019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:13:09.963202 3142 tablet_service.cc:1467] Processing CreateTablet for tablet c162e6df632241f79393fc7ae1a9cfc5 (DEFAULT_TABLE table=test-table [id=74d01397a9944966b0d5c7afdba01a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:09.963201 3281 tablet_service.cc:1467] Processing CreateTablet for tablet c162e6df632241f79393fc7ae1a9cfc5 (DEFAULT_TABLE table=test-table [id=74d01397a9944966b0d5c7afdba01a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:09.963881 3142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c162e6df632241f79393fc7ae1a9cfc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:09.963866 3281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c162e6df632241f79393fc7ae1a9cfc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:09.966471 3239 tablet_bootstrap.cc:492] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d: Bootstrap starting.
I20260929 03:13:09.967520 3239 tablet_bootstrap.cc:654] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:09.968461 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.962276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:34581 (local address 127.0.0.1:41174)
0929 03:13:09.962870 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:09.962888 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:09.962966 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:09.963688 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:09.963691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:09.963711 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:09.963823 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.963829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.965242 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.965245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.966423 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:09.966466 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.968013 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.968017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:09.968137 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.968160 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:13:09.968238 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:13:09.968327 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514}
I20260929 03:13:09.968463 3100 tablet_bootstrap.cc:492] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: Bootstrap starting.
I20260929 03:13:09.968513 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:09.962605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.20.174.67:34581)
0929 03:13:09.963159 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:09.963171 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:09.963195 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:13:09.963279 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:09.963284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:09.963512 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:09.963702 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:09.963960 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.965081 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.966930 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:09.967880 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:09.968006 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:09.968187 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:09.968280 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 03:13:09.968375 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":123,"threads_started":1}
I20260929 03:13:09.969250 3100 tablet_bootstrap.cc:654] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:09.969282 3239 tablet_bootstrap.cc:492] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d: No bootstrap required, opened a new log
I20260929 03:13:09.969362 3239 ts_tablet_manager.cc:1397] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:13:09.969846 3239 raft_consensus.cc:348] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:09.970013 3239 raft_consensus.cc:374] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:09.970062 3239 raft_consensus.cc:729] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53297af6141d43cda47548443c3dc00d, State: Initialized, Role: FOLLOWER
I20260929 03:13:09.970230 3239 consensus_queue.cc:260] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:09.970562 3239 ts_tablet_manager.cc:1428] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 03:13:09.971114 3100 tablet_bootstrap.cc:492] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: No bootstrap required, opened a new log
I20260929 03:13:09.971243 3100 ts_tablet_manager.cc:1397] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:13:09.971863 3100 raft_consensus.cc:348] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:09.971988 3100 raft_consensus.cc:374] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:09.972041 3100 raft_consensus.cc:729] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0fa121c26942b0b1f012c711241262, State: Initialized, Role: FOLLOWER
I20260929 03:13:09.972191 3100 consensus_queue.cc:260] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:09.972183 3417 tablet_service.cc:1467] Processing CreateTablet for tablet c162e6df632241f79393fc7ae1a9cfc5 (DEFAULT_TABLE table=test-table [id=74d01397a9944966b0d5c7afdba01a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:09.972915 3100 ts_tablet_manager.cc:1428] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:13:09.973382 3417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c162e6df632241f79393fc7ae1a9cfc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:09.981467 3495 tablet_bootstrap.cc:492] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: Bootstrap starting.
I20260929 03:13:09.983980 3495 tablet_bootstrap.cc:654] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:09.985272 3495 log.cc:824] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:09.987620 3495 tablet_bootstrap.cc:492] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: No bootstrap required, opened a new log
I20260929 03:13:09.987902 3495 ts_tablet_manager.cc:1397] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
W20260929 03:13:09.990759 3209 tablet.cc:2369] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:13:09.991688 3495 raft_consensus.cc:348] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:09.991952 3495 raft_consensus.cc:374] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:09.992005 3495 raft_consensus.cc:729] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea91a0fb644441796e207f1e9d8d86d, State: Initialized, Role: FOLLOWER
I20260929 03:13:09.992487 3495 consensus_queue.cc:260] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:09.993247 3483 heartbeater.cc:503] Master 127.20.174.126:45109 was elected leader, sending a full tablet report...
I20260929 03:13:09.993598 3495 ts_tablet_manager.cc:1428] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:13:10.137909 3214 raft_consensus.cc:482] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:13:10.138229 3214 raft_consensus.cc:504] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:10.139528 3214 leader_election.cc:288] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581), 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263)
I20260929 03:13:10.144028 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:10.139984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:50015 (local address 127.20.174.67:34581)
0929 03:13:10.140126 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:10.140131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:10.140464 (+ 333us) server_negotiation.cc:407] Connection header received
0929 03:13:10.140630 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:10.140634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:10.140685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:10.140771 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:10.141063 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:10.141689 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:10.142959 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:10.143388 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:10.143493 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:10.143570 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:10.143796 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 03:13:10.143901 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:13:10.144037 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:10.139846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:34581 (local address 127.20.174.65:50015)
0929 03:13:10.140331 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:10.140347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:10.140476 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:10.140808 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:10.140810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:10.140825 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:10.140949 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:10.140958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:10.141834 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:10.141838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:10.142812 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:10.142823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:10.143527 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:10.143534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:10.143655 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:10.143678 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:13:10.143752 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:13:10.143849 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":66,"thread_start_us":82,"threads_started":1}
I20260929 03:13:10.144110 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:10.139846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:38263 (local address 127.20.174.65:45073)
0929 03:13:10.140195 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:10.140208 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:10.140325 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:10.140661 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:10.140664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:10.140683 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:10.140811 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:10.140819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:10.141850 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:10.141854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:10.142833 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:10.142844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:10.143613 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:10.143617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:10.143794 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:10.143819 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:13:10.143894 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:13:10.143965 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":121,"thread_start_us":75,"threads_started":1}
I20260929 03:13:10.144145 3353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:10.139977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:45073 (local address 127.20.174.66:38263)
0929 03:13:10.140122 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:10.140129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:10.140306 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:13:10.140427 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:10.140433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:10.140514 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:10.140627 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:10.141007 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:10.141669 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:10.142998 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:10.143470 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:10.143595 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:10.143774 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:10.143939 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:13:10.144017 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:13:10.144688 3301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c162e6df632241f79393fc7ae1a9cfc5" candidate_uuid: "3c0fa121c26942b0b1f012c711241262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53297af6141d43cda47548443c3dc00d" is_pre_election: true
I20260929 03:13:10.144734 3437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c162e6df632241f79393fc7ae1a9cfc5" candidate_uuid: "3c0fa121c26942b0b1f012c711241262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ea91a0fb644441796e207f1e9d8d86d" is_pre_election: true
I20260929 03:13:10.145037 3437 raft_consensus.cc:2454] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0fa121c26942b0b1f012c711241262 in term 0.
I20260929 03:13:10.145042 3301 raft_consensus.cc:2454] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0fa121c26942b0b1f012c711241262 in term 0.
I20260929 03:13:10.145550 3096 leader_election.cc:302] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c0fa121c26942b0b1f012c711241262, 3ea91a0fb644441796e207f1e9d8d86d, 53297af6141d43cda47548443c3dc00d; no voters:
I20260929 03:13:10.145895 3214 raft_consensus.cc:2781] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:13:10.145978 3214 raft_consensus.cc:482] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:13:10.146030 3214 raft_consensus.cc:3037] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:10.147222 3214 raft_consensus.cc:504] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:10.147615 3214 leader_election.cc:288] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 1 election: Requested vote from peers 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581), 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263)
I20260929 03:13:10.148059 3437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c162e6df632241f79393fc7ae1a9cfc5" candidate_uuid: "3c0fa121c26942b0b1f012c711241262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ea91a0fb644441796e207f1e9d8d86d"
I20260929 03:13:10.148059 3301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c162e6df632241f79393fc7ae1a9cfc5" candidate_uuid: "3c0fa121c26942b0b1f012c711241262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53297af6141d43cda47548443c3dc00d"
I20260929 03:13:10.148198 3437 raft_consensus.cc:3037] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:10.148198 3301 raft_consensus.cc:3037] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:10.149567 3437 raft_consensus.cc:2454] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0fa121c26942b0b1f012c711241262 in term 1.
I20260929 03:13:10.149566 3301 raft_consensus.cc:2454] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0fa121c26942b0b1f012c711241262 in term 1.
I20260929 03:13:10.150098 3097 leader_election.cc:302] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c0fa121c26942b0b1f012c711241262, 3ea91a0fb644441796e207f1e9d8d86d, 53297af6141d43cda47548443c3dc00d; no voters:
I20260929 03:13:10.150357 3214 raft_consensus.cc:2781] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:13:10.150592 3214 raft_consensus.cc:686] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 LEADER]: Becoming Leader. State: Replica: 3c0fa121c26942b0b1f012c711241262, State: Running, Role: LEADER
I20260929 03:13:10.150935 3214 consensus_queue.cc:237] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:10.154208 3020 catalog_manager.cc:5495] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c0fa121c26942b0b1f012c711241262 (127.20.174.65). New cstate: current_term: 1 leader_uuid: "3c0fa121c26942b0b1f012c711241262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } health_report { overall_health: UNKNOWN } } }
W20260929 03:13:10.175285 3484 tablet.cc:2369] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:13:10.211120 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:10.206615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.20.174.65:33967)
0929 03:13:10.207002 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:10.207013 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:10.207028 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:13:10.207117 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:10.207120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:10.207180 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:10.207301 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:13:10.207650 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:10.208312 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:10.209262 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:10.209539 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:10.209628 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:10.210035 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:13:10.210547 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:13:10.210647 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:10.210776 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 03:13:10.210834 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":140,"threads_started":1}
I20260929 03:13:10.215715 3437 raft_consensus.cc:1261] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d [term 1 FOLLOWER]: Refusing update from remote peer 3c0fa121c26942b0b1f012c711241262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:10.215711 3301 raft_consensus.cc:1261] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d [term 1 FOLLOWER]: Refusing update from remote peer 3c0fa121c26942b0b1f012c711241262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:10.216490 3500 consensus_queue.cc:1035] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:13:10.216675 3214 consensus_queue.cc:1035] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:13:10.218317 3348 tablet.cc:2369] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:13:10.233214 3020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:13:10.233410 3020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:13:10.235967 3020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Sending DeleteTablet for 3 replicas of tablet c162e6df632241f79393fc7ae1a9cfc5
I20260929 03:13:10.236724 3142 tablet_service.cc:1514] Processing DeleteTablet for tablet c162e6df632241f79393fc7ae1a9cfc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:10 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:33102
I20260929 03:13:10.236749 3281 tablet_service.cc:1514] Processing DeleteTablet for tablet c162e6df632241f79393fc7ae1a9cfc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:10 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:50796
I20260929 03:13:10.236760 3417 tablet_service.cc:1514] Processing DeleteTablet for tablet c162e6df632241f79393fc7ae1a9cfc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:10 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:41174
I20260929 03:13:10.236924 3354 tablet_replica.cc:330] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d: stopping tablet replica
I20260929 03:13:10.237057 3354 raft_consensus.cc:2229] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:13:10.237210 3513 tablet_replica.cc:330] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: stopping tablet replica
I20260929 03:13:10.237217 3354 raft_consensus.cc:2258] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:10.237352 3513 raft_consensus.cc:2229] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:13:10.237720 3513 raft_consensus.cc:2258] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:10.237952 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 3058
I20260929 03:13:10.238246 3354 ts_tablet_manager.cc:1905] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:13:10.239852 3514 tablet_replica.cc:330] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: stopping tablet replica
I20260929 03:13:10.240331 3514 raft_consensus.cc:2229] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:13:10.240685 3514 raft_consensus.cc:2258] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:10.241639 3354 ts_tablet_manager.cc:1918] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:13:10.241721 3354 log.cc:1196] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/c162e6df632241f79393fc7ae1a9cfc5
I20260929 03:13:10.241973 3354 ts_tablet_manager.cc:1939] T c162e6df632241f79393fc7ae1a9cfc5 P 53297af6141d43cda47548443c3dc00d: Deleting consensus metadata
I20260929 03:13:10.242306 3514 ts_tablet_manager.cc:1905] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:13:10.243225 3007 catalog_manager.cc:4841] TS 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263): tablet c162e6df632241f79393fc7ae1a9cfc5 (table test-table [id=74d01397a9944966b0d5c7afdba01a63]) successfully deleted
I20260929 03:13:10.244704 3514 ts_tablet_manager.cc:1918] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:13:10.244848 3514 log.cc:1196] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/c162e6df632241f79393fc7ae1a9cfc5
I20260929 03:13:10.245239 3514 ts_tablet_manager.cc:1939] T c162e6df632241f79393fc7ae1a9cfc5 P 3ea91a0fb644441796e207f1e9d8d86d: Deleting consensus metadata
W20260929 03:13:10.245275 3005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:33967 (error 108)
I20260929 03:13:10.245581 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 3212
I20260929 03:13:10.246479 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:10.246147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:33967 (local address 127.0.0.1:33122)
0929 03:13:10.246279 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:10.246353 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:33967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 03:13:10.247153 3006 catalog_manager.cc:4841] TS 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581): tablet c162e6df632241f79393fc7ae1a9cfc5 (table test-table [id=74d01397a9944966b0d5c7afdba01a63]) successfully deleted
W20260929 03:13:10.247390 3005 catalog_manager.cc:4584] TS 3c0fa121c26942b0b1f012c711241262 (127.20.174.65:33967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c162e6df632241f79393fc7ae1a9cfc5: Network error: Client connection negotiation failed: client connection to 127.20.174.65:33967: connect: Connection refused (error 111)
I20260929 03:13:10.252588 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 3351
I20260929 03:13:10.258037 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 2987
I20260929 03:13:10.264663 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:45109
--webserver_interface=127.20.174.126
--webserver_port=41371
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:45109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:10.380191 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:10.380537 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:10.380615 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:10.385454 3516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:13:10.385555 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:10.385592 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:10.385632 3516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:13:10.385668 3516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:13:10.391492 3516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:45109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:45109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=41371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:10.392675 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:10.393843 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:10.403816 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:10.403817 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:10.404072 3516 server_base.cc:1034] running on GCE node
W20260929 03:13:10.403816 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:10.404945 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:10.406150 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:10.407380 3516 hybrid_clock.cc:648] HybridClock initialized: now 1790651590407359 us; error 36 us; skew 500 ppm
Sep 29 03:13:10 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651590, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:10.410898 3516 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:13:10.411999 3516 webserver.cc:460] Webserver started at http://127.20.174.126:41371/ using document root <none> and password file <none>
I20260929 03:13:10.412635 3516 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:10.412804 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:10.417062 3516 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:13:10.420011 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:10.421453 3516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 03:13:10.421627 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "44219ceaaceb4843a5c3449b4a40a166"
format_stamp: "Formatted at 2026-09-29 03:13:07 on dist-test-slave-k1dg"
server_key: "b19e4851130c38b8e137344cd6991bbb"
server_key_iv: "80a739e3a27b86571ee032eb88c86deb"
server_key_version: "encryptionkey@0"
I20260929 03:13:10.422277 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:10.436110 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:10.439365 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:10.439586 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:10.447219 3516 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:45109
I20260929 03:13:10.447561 3583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:45109 every 8 connection(s)
I20260929 03:13:10.448448 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:13:10.451721 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 3516
I20260929 03:13:10.452198 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:33967
--local_ip_for_outbound_sockets=127.20.174.65
--tserver_master_addrs=127.20.174.126:45109
--webserver_port=36937
--webserver_interface=127.20.174.65
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:13:10.453908 3584 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:13:10.455856 3584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Bootstrap starting.
I20260929 03:13:10.473875 3584 log.cc:824] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:10.485487 3584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:13:10.485937 3584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166: Bootstrap complete.
I20260929 03:13:10.490242 3584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:10.490754 3584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44219ceaaceb4843a5c3449b4a40a166, State: Initialized, Role: FOLLOWER
I20260929 03:13:10.491285 3584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:10.491410 3584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:13:10.491511 3584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:13:10.491639 3584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:13:10.493680 3584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:10.494063 3584 leader_election.cc:302] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44219ceaaceb4843a5c3449b4a40a166; no voters:
I20260929 03:13:10.494418 3584 leader_election.cc:288] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:13:10.494545 3588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:13:10.494758 3588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [term 3 LEADER]: Becoming Leader. State: Replica: 44219ceaaceb4843a5c3449b4a40a166, State: Running, Role: LEADER
I20260929 03:13:10.495059 3588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } }
I20260929 03:13:10.495718 3584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:13:10.496709 3590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44219ceaaceb4843a5c3449b4a40a166. Latest consensus state: current_term: 3 leader_uuid: "44219ceaaceb4843a5c3449b4a40a166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } } }
I20260929 03:13:10.496919 3590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:10.497359 3597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:13:10.497354 3589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "44219ceaaceb4843a5c3449b4a40a166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44219ceaaceb4843a5c3449b4a40a166" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 45109 } } }
I20260929 03:13:10.497494 3589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44219ceaaceb4843a5c3449b4a40a166 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:10.502943 3597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:13:10.503671 3597 catalog_manager.cc:1259] Loaded cluster ID: c4eeee4770fd408b95d5bc09355c7dc0
I20260929 03:13:10.503738 3597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 03:13:10.504781 3597 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f74a07849ad google::LogMessage::Fail() at ??:0
@ 0x7f74a078788a google::LogMessage::SendToLog() at ??:0
@ 0x7f74a07844de google::LogMessage::Flush() at ??:0
@ 0x7f74a0785a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f74ab0c3fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f74ab0c4d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f74ab0c1579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f74ab105640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f74a29d63d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f74a29d6c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f74a29d83f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f74a29c7512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f74a400b6db start_thread at ??:0
@ 0x7f749f28261f clone at ??:0
W20260929 03:13:10.581519 3586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:10.581770 3586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:10.581815 3586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:10.586527 3586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:10.586582 3586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:10.586647 3586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:13:10.593030 3586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:33967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=36937
--tserver_master_addrs=127.20.174.126:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:10.594122 3586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:10.595211 3586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:10.605690 3626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:10.605883 3586 server_base.cc:1034] running on GCE node
W20260929 03:13:10.605695 3628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:10.605731 3625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:10.606599 3586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:10.607357 3586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:10.608582 3586 hybrid_clock.cc:648] HybridClock initialized: now 1790651590608541 us; error 59 us; skew 500 ppm
Sep 29 03:13:10 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651590, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:10.611654 3586 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:13:10.612639 3586 webserver.cc:460] Webserver started at http://127.20.174.65:36937/ using document root <none> and password file <none>
I20260929 03:13:10.613250 3586 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:10.613355 3586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:10.617714 3586 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 03:13:10.620286 3635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:10.621647 3586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:13:10.621799 3586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "3c0fa121c26942b0b1f012c711241262"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "d43db2b657743c4d12393f6e72551c05"
server_key_iv: "2136042d05f9d10a6f7c92d30267c2b7"
server_key_version: "encryptionkey@0"
I20260929 03:13:10.622380 3586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:10.641980 3586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:10.645571 3586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:10.645807 3586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:10.646514 3586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:10.647766 3642 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:13:10.654871 3586 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:13:10.654994 3586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260929 03:13:10.655086 3586 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:13:10.656836 3586 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:13:10.656915 3586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 03:13:10.657341 3642 tablet_bootstrap.cc:492] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: Bootstrap starting.
I20260929 03:13:10.668273 3586 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:33967
I20260929 03:13:10.668466 3749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:33967 every 8 connection(s)
I20260929 03:13:10.669392 3586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:13:10.669804 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 3586
I20260929 03:13:10.670326 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:38263
--local_ip_for_outbound_sockets=127.20.174.66
--tserver_master_addrs=127.20.174.126:45109
--webserver_port=36831
--webserver_interface=127.20.174.66
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:13:10.678236 3642 log.cc:824] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:10.685292 3642 tablet_bootstrap.cc:492] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:13:10.685686 3642 tablet_bootstrap.cc:492] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: Bootstrap complete.
I20260929 03:13:10.686937 3642 ts_tablet_manager.cc:1397] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.007s
I20260929 03:13:10.689553 3642 raft_consensus.cc:348] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:10.689992 3642 raft_consensus.cc:729] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0fa121c26942b0b1f012c711241262, State: Initialized, Role: FOLLOWER
I20260929 03:13:10.690549 3642 consensus_queue.cc:260] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:10.691460 3642 ts_tablet_manager.cc:1428] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260929 03:13:10.817421 3753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:13:10.671880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:57189)
0929 03:13:10.672584 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:10.672652 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:10.673800 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:10.816612 (+142812us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":184,"threads_started":1}
W20260929 03:13:10.818085 3750 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:45109 (0 consecutive failures): Network error: Failed to ping master at 127.20.174.126:45109: Client connection negotiation failed: client connection to 127.20.174.126:45109: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:13:10.818988 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:10.818655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:44335)
0929 03:13:10.818794 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:10.818877 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 03:13:10.819859 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:10.819602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:58247)
0929 03:13:10.819732 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:10.819788 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 03:13:10.820099 3750 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:13:10.831738 3752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:10.832006 3752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:10.832088 3752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:10.836313 3752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:10.836373 3752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:10.836480 3752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:13:10.841977 3752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:38263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=36831
--tserver_master_addrs=127.20.174.126:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:10.843132 3752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:10.844151 3752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:10.854041 3763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:10.854204 3752 server_base.cc:1034] running on GCE node
W20260929 03:13:10.854082 3761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:10.854115 3760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:10.854815 3752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:10.855520 3752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:10.856694 3752 hybrid_clock.cc:648] HybridClock initialized: now 1790651590856678 us; error 32 us; skew 500 ppm
Sep 29 03:13:10 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651590, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:10.859673 3752 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:13:10.860522 3752 webserver.cc:460] Webserver started at http://127.20.174.66:36831/ using document root <none> and password file <none>
I20260929 03:13:10.861042 3752 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:10.861104 3752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:10.865072 3752 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:13:10.867349 3770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:10.868745 3752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:10.868907 3752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "53297af6141d43cda47548443c3dc00d"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "11cee5a9f76d7b280723e5f1394d893a"
server_key_iv: "14ede0069e2b5fe8f4030b4a0b96f977"
server_key_version: "encryptionkey@0"
I20260929 03:13:10.869525 3752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:10.893000 3752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:10.896799 3752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:10.897047 3752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:10.897747 3752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:10.898742 3752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:13:10.898816 3752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:10.898890 3752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:13:10.898942 3752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:10.908427 3752 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:38263
I20260929 03:13:10.908442 3883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:38263 every 8 connection(s)
I20260929 03:13:10.909510 3752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:13:10.910015 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 3752
I20260929 03:13:10.910565 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:34581
--local_ip_for_outbound_sockets=127.20.174.67
--tserver_master_addrs=127.20.174.126:45109
--webserver_port=38757
--webserver_interface=127.20.174.67
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:13:10.914114 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:10.912095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:56621)
0929 03:13:10.912660 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:10.912842 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":117,"threads_started":1}
W20260929 03:13:10.914651 3884 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:45109 (0 consecutive failures): Network error: Failed to ping master at 127.20.174.126:45109: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
I20260929 03:13:10.915357 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:10.915041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:37453)
0929 03:13:10.915191 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:10.915261 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 03:13:10.916054 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:10.915792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:33245)
0929 03:13:10.915948 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:10.916002 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 03:13:10.916297 3884 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:13:11.030560 3886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:11.030900 3886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:11.030993 3886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:11.035073 3886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:11.035185 3886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:11.035281 3886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:13:11.042287 3886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:34581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=38757
--tserver_master_addrs=127.20.174.126:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:11.043540 3886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:11.044628 3886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:13:11.055321 3886 server_base.cc:1034] running on GCE node
W20260929 03:13:11.055262 3893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:11.055269 3892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:11.055335 3895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:11.056046 3886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:11.056762 3886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:11.057976 3886 hybrid_clock.cc:648] HybridClock initialized: now 1790651591057950 us; error 33 us; skew 500 ppm
Sep 29 03:13:11 dist-test-slave-k1dg krb5kdc[2439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651591, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:11.060837 3886 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:13:11.061735 3886 webserver.cc:460] Webserver started at http://127.20.174.67:38757/ using document root <none> and password file <none>
I20260929 03:13:11.062307 3886 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:11.062423 3886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:11.066399 3886 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:13:11.068624 3902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:11.070003 3886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:13:11.070156 3886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "3ea91a0fb644441796e207f1e9d8d86d"
format_stamp: "Formatted at 2026-09-29 03:13:08 on dist-test-slave-k1dg"
server_key: "17d4134ebfb91bceaeac76327ec40e2e"
server_key_iv: "5e8f41697a80004abd6061bc84bf58cd"
server_key_version: "encryptionkey@0"
I20260929 03:13:11.070802 3886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:11.091600 3886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:11.094818 3886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:11.095053 3886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:11.095688 3886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:11.096549 3886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:13:11.096619 3886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:11.096692 3886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:13:11.096732 3886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:11.106144 3886 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:34581
I20260929 03:13:11.106169 4015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:34581 every 8 connection(s)
I20260929 03:13:11.107277 3886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:13:11.109736 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 3886
I20260929 03:13:11.111608 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:11.109518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:57961)
0929 03:13:11.110100 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:11.110251 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":111,"threads_started":1}
W20260929 03:13:11.112056 4016 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:45109 (0 consecutive failures): Network error: Failed to ping master at 127.20.174.126:45109: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
I20260929 03:13:11.112701 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:11.112474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:34825)
0929 03:13:11.112590 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:11.112648 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 03:13:11.113292 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:11.113088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:52211)
0929 03:13:11.113188 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:11.113235 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 03:13:11.113466 4016 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:13:11.821179 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:11.820744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:54255)
0929 03:13:11.821010 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:11.821068 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
I20260929 03:13:11.917434 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:11.916948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:38819)
0929 03:13:11.917246 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:11.917325 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260929 03:13:12.007526 4022 raft_consensus.cc:482] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:13:12.007781 4022 raft_consensus.cc:504] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:12.008924 4022 leader_election.cc:288] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581), 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263)
Sep 29 03:13:12 dist-test-slave-k1dg krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651590, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.66@KRBTEST.COM
Sep 29 03:13:12 dist-test-slave-k1dg krb5kdc[2439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651590, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.67@KRBTEST.COM
I20260929 03:13:12.024658 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:12.009217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:34581 (local address 127.20.174.65:51245)
0929 03:13:12.009356 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:12.009373 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:12.009488 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:12.011497 (+ 2009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:12.011504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:12.012098 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:12.012585 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:12.012602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:12.014085 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:12.014089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:12.015087 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:12.015098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:12.015880 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:12.015884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:12.016034 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:12.016951 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:12.016965 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:12.019276 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:12.021694 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:12.021697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:12.021698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:12.022097 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:12.022612 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:12.022615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:12.022616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:12.022705 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:12.023532 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:12.023544 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:12.023821 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 03:13:12.024018 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 03:13:12.024421 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":181,"spinlock_wait_cycles":11264}
I20260929 03:13:12.024686 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:12.009441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:51245 (local address 127.20.174.67:34581)
0929 03:13:12.009920 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:12.009932 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:12.009970 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:13:12.010766 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:12.010773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:12.011113 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:12.011453 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:12.012804 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:12.013910 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:12.015234 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:12.015741 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:12.015931 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:12.019614 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:12.019619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:12.019632 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:12.019663 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:12.021520 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:12.022298 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:12.022302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:12.022307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:12.022503 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:12.022843 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:12.022846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:12.022847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:12.023300 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:12.023558 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:12.024160 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 03:13:12.024486 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":87,"threads_started":1}
I20260929 03:13:12.024689 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:12.009441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:51673 (local address 127.20.174.66:38263)
0929 03:13:12.009783 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:12.009811 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:12.009870 (+ 59us) server_negotiation.cc:407] Connection header received
0929 03:13:12.010712 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:12.010718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:12.011097 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:12.011424 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:12.012778 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:12.013789 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:12.015221 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:12.015719 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:12.015930 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:12.019176 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:12.019187 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:12.019210 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:12.019257 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:12.021488 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:12.022160 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:12.022169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:12.022176 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:12.022363 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:12.022812 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:12.022818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:12.022821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:12.023301 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:12.023459 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:12.024165 (+ 706us) server_negotiation.cc:299] Negotiation successful
0929 03:13:12.024463 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260929 03:13:12.024682 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:12.009213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:38263 (local address 127.20.174.65:51673)
0929 03:13:12.009679 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:12.009693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:12.009779 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:12.011497 (+ 1718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:12.011504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:12.012099 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:12.012585 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:12.012602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:12.013967 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:12.013971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:12.015086 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:12.015098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:12.015860 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:12.015864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:12.016030 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:12.016872 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:12.016891 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:12.018879 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:12.021639 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:12.021647 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:12.021665 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:12.022026 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:12.022511 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:12.022514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:12.022516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:12.022701 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:12.023529 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:12.023536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:12.023821 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 03:13:12.024018 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 03:13:12.024507 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":529,"spinlock_wait_cycles":3840,"thread_start_us":88,"threads_started":1}
W20260929 03:13:12.028765 3639 leader_election.cc:341] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263): Not found: Tablet not found: c162e6df632241f79393fc7ae1a9cfc5
W20260929 03:13:12.028949 3638 leader_election.cc:341] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581): Not found: Tablet not found: c162e6df632241f79393fc7ae1a9cfc5
I20260929 03:13:12.029089 3638 leader_election.cc:302] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c0fa121c26942b0b1f012c711241262; no voters: 3ea91a0fb644441796e207f1e9d8d86d, 53297af6141d43cda47548443c3dc00d
I20260929 03:13:12.029382 4022 raft_consensus.cc:2726] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:13:12.114662 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:12.114154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:41127)
0929 03:13:12.114492 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:12.114561 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260929 03:13:12.822758 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:12.822136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:44371)
0929 03:13:12.822592 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:12.822654 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":136,"threads_started":1}
I20260929 03:13:12.918851 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:12.918258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:54093)
0929 03:13:12.918660 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:12.918745 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":95,"threads_started":1}
I20260929 03:13:13.116107 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:13.115571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:49697)
0929 03:13:13.115922 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:13.115992 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":75,"threads_started":1}
I20260929 03:13:13.824049 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:13.823595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:45053)
0929 03:13:13.823876 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:13.823940 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
I20260929 03:13:13.920483 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:13.919913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:33039)
0929 03:13:13.920270 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:13.920364 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":93,"threads_started":1}
I20260929 03:13:14.117542 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:14.117092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:50081)
0929 03:13:14.117384 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:14.117446 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260929 03:13:14.163962 4033 raft_consensus.cc:482] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:13:14.164122 4033 raft_consensus.cc:504] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:14.164773 4033 leader_election.cc:288] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581), 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263)
W20260929 03:13:14.165558 3639 leader_election.cc:341] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263): Not found: Tablet not found: c162e6df632241f79393fc7ae1a9cfc5
W20260929 03:13:14.165700 3638 leader_election.cc:341] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581): Not found: Tablet not found: c162e6df632241f79393fc7ae1a9cfc5
I20260929 03:13:14.165796 3638 leader_election.cc:302] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c0fa121c26942b0b1f012c711241262; no voters: 3ea91a0fb644441796e207f1e9d8d86d, 53297af6141d43cda47548443c3dc00d
I20260929 03:13:14.165974 4033 raft_consensus.cc:2726] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:13:14.825497 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:14.824985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:54899)
0929 03:13:14.825260 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:14.825326 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260929 03:13:14.921787 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:14.921355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:55663)
0929 03:13:14.921636 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:14.921694 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
I20260929 03:13:15.119103 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:15.118614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:51147)
0929 03:13:15.118938 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:15.119005 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260929 03:13:15.827373 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:15.826914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:38891)
0929 03:13:15.827209 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:15.827273 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":72,"threads_started":1}
I20260929 03:13:15.923237 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:15.922771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:35131)
0929 03:13:15.923057 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:15.923123 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":76,"threads_started":1}
I20260929 03:13:16.120656 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:16.120045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:34573)
0929 03:13:16.120467 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:16.120548 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":111,"threads_started":1}
I20260929 03:13:16.495151 4040 raft_consensus.cc:482] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:13:16.495306 4040 raft_consensus.cc:504] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:16.495860 4040 leader_election.cc:288] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581), 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263)
W20260929 03:13:16.496865 3638 leader_election.cc:341] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581): Not found: Tablet not found: c162e6df632241f79393fc7ae1a9cfc5
W20260929 03:13:16.496985 3639 leader_election.cc:341] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263): Not found: Tablet not found: c162e6df632241f79393fc7ae1a9cfc5
I20260929 03:13:16.497098 3639 leader_election.cc:302] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c0fa121c26942b0b1f012c711241262; no voters: 3ea91a0fb644441796e207f1e9d8d86d, 53297af6141d43cda47548443c3dc00d
I20260929 03:13:16.497306 4040 raft_consensus.cc:2726] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:13:16.828774 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:16.828317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:41113)
0929 03:13:16.828597 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:16.828667 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
I20260929 03:13:16.924996 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:16.924482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:35737)
0929 03:13:16.924824 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:16.924896 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":79,"threads_started":1}
I20260929 03:13:17.122066 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:17.121605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:57507)
0929 03:13:17.121885 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:17.121962 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260929 03:13:17.830261 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:17.829670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:55719)
0929 03:13:17.830084 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:17.830156 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":212,"threads_started":1}
I20260929 03:13:17.926687 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:17.926116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:44899)
0929 03:13:17.926517 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:17.926587 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":101,"threads_started":1}
I20260929 03:13:18.123770 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:18.123197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:38137)
0929 03:13:18.123599 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:18.123666 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":114,"threads_started":1}
I20260929 03:13:18.831921 4047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:18.831377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:38483)
0929 03:13:18.831661 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:18.831739 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
I20260929 03:13:18.928151 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:18.927625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:34749)
0929 03:13:18.927995 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:18.928054 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":156,"threads_started":1}
I20260929 03:13:19.125394 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:19.124733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:52961)
0929 03:13:19.125073 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:19.125250 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":124,"threads_started":1}
I20260929 03:13:19.833292 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:19.832832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:43813)
0929 03:13:19.833126 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:19.833189 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":76,"threads_started":1}
I20260929 03:13:19.929618 4051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:19.929187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:35223)
0929 03:13:19.929467 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:19.929523 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20260929 03:13:20.127110 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:20.126626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:34729)
0929 03:13:20.126946 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:20.127011 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":84,"threads_started":1}
I20260929 03:13:20.233670 4053 raft_consensus.cc:482] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:13:20.233821 4053 raft_consensus.cc:504] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:20.234421 4053 leader_election.cc:288] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581), 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263)
W20260929 03:13:20.235263 3638 leader_election.cc:341] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581): Not found: Tablet not found: c162e6df632241f79393fc7ae1a9cfc5
W20260929 03:13:20.235388 3639 leader_election.cc:341] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263): Not found: Tablet not found: c162e6df632241f79393fc7ae1a9cfc5
I20260929 03:13:20.235519 3639 leader_election.cc:302] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c0fa121c26942b0b1f012c711241262; no voters: 3ea91a0fb644441796e207f1e9d8d86d, 53297af6141d43cda47548443c3dc00d
I20260929 03:13:20.235745 4053 raft_consensus.cc:2726] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:13:20.834807 4054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:20.834260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:59525)
0929 03:13:20.834624 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:20.834690 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":91,"threads_started":1}
I20260929 03:13:20.930994 4055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:20.930539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:56887)
0929 03:13:20.930798 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:20.930857 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260929 03:13:21.128677 4056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:21.128043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:56027)
0929 03:13:21.128466 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:21.128555 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":161,"threads_started":1}
I20260929 03:13:21.836170 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:21.835695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:43755)
0929 03:13:21.835984 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:21.836058 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
I20260929 03:13:21.932370 4058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:21.931919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:60999)
0929 03:13:21.932194 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:21.932267 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260929 03:13:22.130237 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:22.129770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:56447)
0929 03:13:22.130068 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:22.130134 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260929 03:13:22.837603 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:22.837111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:38425)
0929 03:13:22.837403 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:22.837486 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":79,"threads_started":1}
I20260929 03:13:22.933866 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:22.933285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:58619)
0929 03:13:22.933609 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:22.933685 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":83,"threads_started":1}
I20260929 03:13:23.131631 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:23.131160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:50151)
0929 03:13:23.131460 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:23.131526 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":76,"threads_started":1}
I20260929 03:13:23.550233 4063 raft_consensus.cc:482] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:13:23.550366 4063 raft_consensus.cc:504] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea91a0fb644441796e207f1e9d8d86d" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "3c0fa121c26942b0b1f012c711241262" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 33967 } } peers { permanent_uuid: "53297af6141d43cda47548443c3dc00d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 38263 } }
I20260929 03:13:23.550879 4063 leader_election.cc:288] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581), 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263)
W20260929 03:13:23.551685 3638 leader_election.cc:341] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ea91a0fb644441796e207f1e9d8d86d (127.20.174.67:34581): Not found: Tablet not found: c162e6df632241f79393fc7ae1a9cfc5
W20260929 03:13:23.551795 3639 leader_election.cc:341] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53297af6141d43cda47548443c3dc00d (127.20.174.66:38263): Not found: Tablet not found: c162e6df632241f79393fc7ae1a9cfc5
I20260929 03:13:23.551954 3639 leader_election.cc:302] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c0fa121c26942b0b1f012c711241262; no voters: 3ea91a0fb644441796e207f1e9d8d86d, 53297af6141d43cda47548443c3dc00d
I20260929 03:13:23.552124 4063 raft_consensus.cc:2726] T c162e6df632241f79393fc7ae1a9cfc5 P 3c0fa121c26942b0b1f012c711241262 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:13:23.839076 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:23.838602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:47335)
0929 03:13:23.838887 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:23.838951 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260929 03:13:23.935406 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:23.934961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:36659)
0929 03:13:23.935227 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:23.935291 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260929 03:13:24.133111 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:24.132608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:58483)
0929 03:13:24.132929 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:24.132998 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":84,"threads_started":1}
I20260929 03:13:24.840587 4067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:24.840093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:58467)
0929 03:13:24.840397 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:24.840474 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
I20260929 03:13:24.936878 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:24.936378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:57093)
0929 03:13:24.936715 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:24.936778 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":144,"threads_started":1}
I20260929 03:13:25.134742 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:25.134217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.67:58091)
0929 03:13:25.134577 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:25.134642 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":96,"threads_started":1}
I20260929 03:13:25.842119 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:25.841683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.65:40751)
0929 03:13:25.841960 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:25.842022 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":73,"threads_started":1}
I20260929 03:13:25.938421 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:25.937991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:45109 (local address 127.20.174.66:38745)
0929 03:13:25.938251 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:25.938318 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260929 03:13:26.116798 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 3586
I20260929 03:13:26.122821 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 3752
I20260929 03:13:26.128247 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 3886
2026-09-29T03:13:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19661 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:13:26 dist-test-slave-k1dg krb5kdc[4074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:13:26 dist-test-slave-k1dg krb5kdc[4074](info): set up 2 sockets
Sep 29 03:13:26 dist-test-slave-k1dg krb5kdc[4074](info): commencing operation
krb5kdc: starting...
W20260929 03:13:27.214937 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:13:27 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:13:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:13:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:13:27.342309 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:37661
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:37661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:27.456862 4087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:27.457248 4087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:27.457338 4087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:27.461619 4087 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:13:27.461727 4087 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:13:27.461755 4087 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:13:27.461779 4087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:13:27.461820 4087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:27.461863 4087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:27.461920 4087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:13:27.461961 4087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:13:27.467690 4087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:37661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:37661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:27.468907 4087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:27.470010 4087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:27.478955 4095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:27.478967 4093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:27.478967 4092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:27.479410 4087 server_base.cc:1034] running on GCE node
I20260929 03:13:27.480186 4087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:27.481551 4087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:27.482831 4087 hybrid_clock.cc:648] HybridClock initialized: now 1790651607482789 us; error 67 us; skew 500 ppm
Sep 29 03:13:27 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651607, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:27.486124 4087 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:13:27.487257 4087 webserver.cc:460] Webserver started at http://127.20.174.126:45619/ using document root <none> and password file <none>
I20260929 03:13:27.487977 4087 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:27.488085 4087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:27.488348 4087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:27.490793 4087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21"
format_stamp: "Formatted at 2026-09-29 03:13:27 on dist-test-slave-k1dg"
server_key: "bdb9e94d2387baced4903a0be9c349f5"
server_key_iv: "6b4f1a11691ebf8a8b001302e2650a4c"
server_key_version: "encryptionkey@0"
I20260929 03:13:27.491488 4087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21"
format_stamp: "Formatted at 2026-09-29 03:13:27 on dist-test-slave-k1dg"
server_key: "bdb9e94d2387baced4903a0be9c349f5"
server_key_iv: "6b4f1a11691ebf8a8b001302e2650a4c"
server_key_version: "encryptionkey@0"
I20260929 03:13:27.496138 4087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:13:27.499008 4102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:27.500653 4087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:13:27.500844 4087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21"
format_stamp: "Formatted at 2026-09-29 03:13:27 on dist-test-slave-k1dg"
server_key: "bdb9e94d2387baced4903a0be9c349f5"
server_key_iv: "6b4f1a11691ebf8a8b001302e2650a4c"
server_key_version: "encryptionkey@0"
I20260929 03:13:27.500983 4087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:27.539168 4087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:27.542956 4087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:27.543215 4087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:27.551134 4087 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:37661
I20260929 03:13:27.551503 4154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:37661 every 8 connection(s)
I20260929 03:13:27.552289 4087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:13:27.555722 4155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:27.559767 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 4087
I20260929 03:13:27.559935 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:13:27.560276 21177 external_mini_cluster.cc:1428] Setting key 9793c36709ad90e4feba1021c3e963df
I20260929 03:13:27.562055 4155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Bootstrap starting.
I20260929 03:13:27.564605 4155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:27.565668 4155 log.cc:824] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:27.568091 4155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: No bootstrap required, opened a new log
Sep 29 03:13:27 dist-test-slave-k1dg krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:27.573161 4155 raft_consensus.cc:348] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:13:27.573477 4155 raft_consensus.cc:374] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:27.573537 4155 raft_consensus.cc:729] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d389bb2c24cb4471b8ffbeaa0bc93b21, State: Initialized, Role: FOLLOWER
I20260929 03:13:27.574115 4155 consensus_queue.cc:260] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:13:27.574301 4155 raft_consensus.cc:388] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:13:27.574388 4155 raft_consensus.cc:482] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:13:27.574527 4155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:27.575832 4155 raft_consensus.cc:504] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:13:27.576224 4155 leader_election.cc:302] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d389bb2c24cb4471b8ffbeaa0bc93b21; no voters:
I20260929 03:13:27.576578 4155 leader_election.cc:288] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:13:27.576737 4160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:13:27.576989 4160 raft_consensus.cc:686] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 1 LEADER]: Becoming Leader. State: Replica: d389bb2c24cb4471b8ffbeaa0bc93b21, State: Running, Role: LEADER
I20260929 03:13:27.577371 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:27.561771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.20.174.126:37661)
0929 03:13:27.562332 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:27.562349 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:27.562388 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:13:27.563143 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:27.563164 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:27.563579 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:27.563972 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:27.564936 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:27.566401 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:27.567489 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:27.568219 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:27.568384 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:27.571468 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:27.571478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:27.571505 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:27.571552 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:27.573775 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:27.574395 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:27.574403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:27.574411 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:27.574599 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:27.574916 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:27.574920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:27.574921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:27.575486 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:27.575691 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:27.575950 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 03:13:27.576290 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":147,"threads_started":1}
I20260929 03:13:27.577888 4155 sys_catalog.cc:564] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:13:27.577368 4160 consensus_queue.cc:237] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:13:27.580418 4162 sys_catalog.cc:455] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } } }
I20260929 03:13:27.580376 4161 sys_catalog.cc:455] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d389bb2c24cb4471b8ffbeaa0bc93b21. Latest consensus state: current_term: 1 leader_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } } }
I20260929 03:13:27.580541 4161 sys_catalog.cc:458] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:27.580541 4162 sys_catalog.cc:458] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:27.581051 4169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:13:27.583979 4169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:13:27.591128 4169 catalog_manager.cc:1347] Generated new cluster ID: 46e16f05f81a41329d44ce3316cf4ee0
I20260929 03:13:27.591230 4169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:13:27.625098 4169 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:13:27.626132 4169 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:13:27.636652 4169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Generated new TSK 0
I20260929 03:13:27.637511 4169 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:13:27.670660 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:37661
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:27.792266 4181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:27.792558 4181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:27.792604 4181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:27.796918 4181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:27.796985 4181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:27.797045 4181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:13:27.802564 4181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:27.803608 4181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:27.804723 4181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:27.815160 4186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:27.815248 4181 server_base.cc:1034] running on GCE node
W20260929 03:13:27.815160 4187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:27.815197 4189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:27.815953 4181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:27.816895 4181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:27.818120 4181 hybrid_clock.cc:648] HybridClock initialized: now 1790651607818099 us; error 34 us; skew 500 ppm
Sep 29 03:13:27 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651607, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:27.821593 4181 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:13:27.823107 4181 webserver.cc:460] Webserver started at http://127.20.174.65:34011/ using document root <none> and password file <none>
I20260929 03:13:27.823762 4181 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:27.823833 4181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:27.824117 4181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:27.826571 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "fd8b6b547d594a72b4f4bcf02e504f4d"
format_stamp: "Formatted at 2026-09-29 03:13:27 on dist-test-slave-k1dg"
server_key: "079d5834a2dc57ac5d60d33145977fd8"
server_key_iv: "091e7f76fa6625b788b491cf7bdcccb7"
server_key_version: "encryptionkey@0"
I20260929 03:13:27.827291 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "fd8b6b547d594a72b4f4bcf02e504f4d"
format_stamp: "Formatted at 2026-09-29 03:13:27 on dist-test-slave-k1dg"
server_key: "079d5834a2dc57ac5d60d33145977fd8"
server_key_iv: "091e7f76fa6625b788b491cf7bdcccb7"
server_key_version: "encryptionkey@0"
I20260929 03:13:27.831477 4181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:13:27.834057 4196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:27.835593 4181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:13:27.835753 4181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "fd8b6b547d594a72b4f4bcf02e504f4d"
format_stamp: "Formatted at 2026-09-29 03:13:27 on dist-test-slave-k1dg"
server_key: "079d5834a2dc57ac5d60d33145977fd8"
server_key_iv: "091e7f76fa6625b788b491cf7bdcccb7"
server_key_version: "encryptionkey@0"
I20260929 03:13:27.835922 4181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:27.847971 4181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:27.851622 4181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:27.851879 4181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:27.852525 4181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:27.853421 4181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:13:27.853499 4181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:27.853570 4181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:13:27.853619 4181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:27.863372 4181 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:35253
I20260929 03:13:27.863390 4309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:35253 every 8 connection(s)
I20260929 03:13:27.864358 4181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:13:27.867803 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 4181
I20260929 03:13:27.867965 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:13:27.868359 21177 external_mini_cluster.cc:1428] Setting key 2db7721e88f67d86774af91b6fbd55f2
Sep 29 03:13:27 dist-test-slave-k1dg krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651607, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:13:27.881120 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:27.866652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:53137 (local address 127.20.174.126:37661)
0929 03:13:27.866804 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:27.866808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:27.867890 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 03:13:27.869560 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:27.869564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:27.869622 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:27.869713 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:27.871294 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:27.871842 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:27.873088 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:27.873333 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:27.873448 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:27.876689 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:27.876693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:27.876696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:27.876732 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:27.878558 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:27.879271 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:27.879274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:27.879275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:27.879400 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:27.879847 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:27.879849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:27.879851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:27.880064 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:27.880147 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:27.880842 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 03:13:27.880988 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:13:27.882130 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:27.867068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:53137)
0929 03:13:27.867606 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:27.867675 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:27.869313 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:27.869964 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:27.869977 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:27.870546 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:27.871109 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:27.871126 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:27.871994 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:27.871997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:27.872934 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:27.872945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:27.873498 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:27.873502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:27.873682 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:27.874407 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:27.874424 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:27.876438 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:27.878695 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:27.878703 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:27.878723 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:27.879152 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:27.879509 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:27.879514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:27.879518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:27.879738 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:27.880164 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:27.880170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:27.880535 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 03:13:27.880747 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 03:13:27.881134 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:13:27.883646 4310 heartbeater.cc:347] Connected to a master server at 127.20.174.126:37661
I20260929 03:13:27.883936 4310 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:27.884514 4310 heartbeater.cc:511] Master 127.20.174.126:37661 requested a full tablet report, sending...
I20260929 03:13:27.886180 4119 ts_manager.cc:194] Registered new tserver with Master: fd8b6b547d594a72b4f4bcf02e504f4d (127.20.174.65:35253)
I20260929 03:13:27.887431 4119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:53137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:13:27.903404 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:37661
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:28.025924 4315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:28.026285 4315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:28.026331 4315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:28.030848 4315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:28.030920 4315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:28.030984 4315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:13:28.036597 4315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:28.037788 4315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:28.038942 4315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:28.050352 4323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:28.050318 4320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:28.050374 4321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:28.050719 4315 server_base.cc:1034] running on GCE node
I20260929 03:13:28.051383 4315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:28.052167 4315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:28.053385 4315 hybrid_clock.cc:648] HybridClock initialized: now 1790651608053370 us; error 33 us; skew 500 ppm
Sep 29 03:13:28 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651608, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:28.056749 4315 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:13:28.057893 4315 webserver.cc:460] Webserver started at http://127.20.174.66:42225/ using document root <none> and password file <none>
I20260929 03:13:28.058650 4315 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:28.058884 4315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:28.059170 4315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:28.061615 4315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "2861338795d74860b8f4ebda76b6a400"
format_stamp: "Formatted at 2026-09-29 03:13:28 on dist-test-slave-k1dg"
server_key: "95f5ca700006673fd7d96e9a8e377414"
server_key_iv: "7b4cde3c8f90618d3f8bf02ab50fa678"
server_key_version: "encryptionkey@0"
I20260929 03:13:28.062359 4315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "2861338795d74860b8f4ebda76b6a400"
format_stamp: "Formatted at 2026-09-29 03:13:28 on dist-test-slave-k1dg"
server_key: "95f5ca700006673fd7d96e9a8e377414"
server_key_iv: "7b4cde3c8f90618d3f8bf02ab50fa678"
server_key_version: "encryptionkey@0"
I20260929 03:13:28.067082 4315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:13:28.069998 4330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:28.071566 4315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:28.071745 4315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "2861338795d74860b8f4ebda76b6a400"
format_stamp: "Formatted at 2026-09-29 03:13:28 on dist-test-slave-k1dg"
server_key: "95f5ca700006673fd7d96e9a8e377414"
server_key_iv: "7b4cde3c8f90618d3f8bf02ab50fa678"
server_key_version: "encryptionkey@0"
I20260929 03:13:28.071902 4315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:28.087152 4315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:28.091212 4315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:28.091441 4315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:28.092093 4315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:28.092994 4315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:13:28.093070 4315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:28.093142 4315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:13:28.093197 4315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:28.103045 4315 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:45887
I20260929 03:13:28.103065 4443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:45887 every 8 connection(s)
I20260929 03:13:28.104072 4315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:13:28.111361 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 4315
I20260929 03:13:28.111528 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:13:28.111822 21177 external_mini_cluster.cc:1428] Setting key bfdfe05a2a2c4d15fdf344b0a41d5e3e
Sep 29 03:13:28 dist-test-slave-k1dg krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651608, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:13:28.122742 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.106587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:60641 (local address 127.20.174.126:37661)
0929 03:13:28.106756 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:28.106761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:28.107749 (+ 988us) server_negotiation.cc:407] Connection header received
0929 03:13:28.109779 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:28.109782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:28.109837 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:28.109923 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:28.111578 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.112278 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.113927 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.114199 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.114343 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.117851 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:28.117855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:28.117857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:28.117888 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:28.119767 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:28.120597 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:28.120601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:28.120602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:28.120698 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:28.121223 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:28.121226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:28.121227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:28.121491 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:28.121581 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:28.122465 (+ 884us) server_negotiation.cc:299] Negotiation successful
0929 03:13:28.122607 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:13:28.123751 4446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.106774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:60641)
0929 03:13:28.107527 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:28.107595 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:28.109480 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:28.110087 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:28.110095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:28.110628 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:28.111325 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.111353 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.112446 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.112456 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.113765 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.113778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.114397 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.114402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.114649 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.115421 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:28.115442 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:28.117582 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:28.119937 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:28.119948 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:28.119971 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:28.120436 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:28.120815 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:28.120819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:28.120821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:28.121066 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:28.121630 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:28.121642 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:28.122054 (+ 412us) client_negotiation.cc:769] Sending connection context
0929 03:13:28.122303 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 03:13:28.122777 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":176,"threads_started":1}
I20260929 03:13:28.125166 4444 heartbeater.cc:347] Connected to a master server at 127.20.174.126:37661
I20260929 03:13:28.125526 4444 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:13:28.126159 4444 heartbeater.cc:511] Master 127.20.174.126:37661 requested a full tablet report, sending...
I20260929 03:13:28.127549 4119 ts_manager.cc:194] Registered new tserver with Master: 2861338795d74860b8f4ebda76b6a400 (127.20.174.66:45887)
I20260929 03:13:28.128197 4119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:60641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:13:28.147013 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:37661
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:28.269686 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:28.270061 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:28.270148 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:28.274718 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:28.274827 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:28.274930 4449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:13:28.280879 4449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:28.282088 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:28.283267 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:28.293579 4457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:28.293581 4455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:28.293561 4454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:28.293975 4449 server_base.cc:1034] running on GCE node
I20260929 03:13:28.294519 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:28.295260 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:28.296442 4449 hybrid_clock.cc:648] HybridClock initialized: now 1790651608296399 us; error 55 us; skew 500 ppm
Sep 29 03:13:28 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651608, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:28.299777 4449 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:13:28.300869 4449 webserver.cc:460] Webserver started at http://127.20.174.67:35377/ using document root <none> and password file <none>
I20260929 03:13:28.301556 4449 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:28.301628 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:28.301869 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:13:28.304204 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "a84bac3401714a35a403c5c6d177dacb"
format_stamp: "Formatted at 2026-09-29 03:13:28 on dist-test-slave-k1dg"
server_key: "a680bd30d5dd4422e86c7486c0130ece"
server_key_iv: "a5579bc8bd436b17370d3cfbd329bde4"
server_key_version: "encryptionkey@0"
I20260929 03:13:28.304821 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "a84bac3401714a35a403c5c6d177dacb"
format_stamp: "Formatted at 2026-09-29 03:13:28 on dist-test-slave-k1dg"
server_key: "a680bd30d5dd4422e86c7486c0130ece"
server_key_iv: "a5579bc8bd436b17370d3cfbd329bde4"
server_key_version: "encryptionkey@0"
I20260929 03:13:28.309127 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 03:13:28.311887 4464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:28.313282 4449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:13:28.313393 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "a84bac3401714a35a403c5c6d177dacb"
format_stamp: "Formatted at 2026-09-29 03:13:28 on dist-test-slave-k1dg"
server_key: "a680bd30d5dd4422e86c7486c0130ece"
server_key_iv: "a5579bc8bd436b17370d3cfbd329bde4"
server_key_version: "encryptionkey@0"
I20260929 03:13:28.313558 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:28.336198 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:28.339865 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:28.340097 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:28.340763 4449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:28.341665 4449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:13:28.341714 4449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:28.341756 4449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:13:28.341771 4449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:28.351780 4449 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:45741
I20260929 03:13:28.351909 4577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:45741 every 8 connection(s)
I20260929 03:13:28.352825 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:13:28.354635 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 4449
I20260929 03:13:28.354774 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:13:28.355122 21177 external_mini_cluster.cc:1428] Setting key 8caa971afff76e08c2465eacea3924e4
Sep 29 03:13:28 dist-test-slave-k1dg krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651608, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:28.368525 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.355148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:46151 (local address 127.20.174.126:37661)
0929 03:13:28.355346 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:28.355349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:28.356387 (+ 1038us) server_negotiation.cc:407] Connection header received
0929 03:13:28.357997 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:28.358000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:28.358056 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:28.358134 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:28.359444 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.359980 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.361194 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.361421 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.361539 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.364261 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:28.364264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:28.364267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:28.364294 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:28.365776 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:28.366314 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:28.366317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:28.366318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:28.366426 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:28.367173 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:28.367176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:28.367177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:28.367382 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:28.367528 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:28.368185 (+ 657us) server_negotiation.cc:299] Negotiation successful
0929 03:13:28.368354 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 03:13:28.369684 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.355637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:46151)
0929 03:13:28.356246 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:28.356294 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:28.357767 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:28.358309 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:28.358317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:28.358776 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:28.359282 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.359297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.360117 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.360121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.361044 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.361054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.361586 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.361590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.361801 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.362480 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:28.362498 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:28.364069 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:28.365910 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:28.365916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:28.365930 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:28.366215 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:28.366708 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:28.366769 (+ 61us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:28.366770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:28.366954 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:28.367507 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:28.367513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:28.367878 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 03:13:28.368132 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 03:13:28.368564 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":186,"threads_started":1}
I20260929 03:13:28.370975 4578 heartbeater.cc:347] Connected to a master server at 127.20.174.126:37661
I20260929 03:13:28.371297 4578 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:28.371858 4578 heartbeater.cc:511] Master 127.20.174.126:37661 requested a full tablet report, sending...
I20260929 03:13:28.373067 4119 ts_manager.cc:194] Registered new tserver with Master: a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741)
I20260929 03:13:28.373699 4119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:46151
I20260929 03:13:28.382064 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:13:28.392809 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.384256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.20.174.126:37661)
0929 03:13:28.384592 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:28.384597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:28.384743 (+ 146us) server_negotiation.cc:407] Connection header received
0929 03:13:28.384909 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:28.384912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:28.384986 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:28.385084 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:28.385826 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.386349 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.387255 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.387487 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.387601 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.389096 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:28.389099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:28.389101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:28.389126 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:28.390898 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:28.391363 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:28.391368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:28.391370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:28.391462 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:28.391759 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:28.391763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:28.391764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:28.392064 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:28.392221 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:28.392497 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 03:13:28.392629 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 03:13:28.396343 4119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:13:28.399801 4119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:13:28.415855 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.409126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:45741 (local address 127.0.0.1:36500)
0929 03:13:28.410037 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:28.410096 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:28.410246 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:28.411086 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:28.411089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:28.411108 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:28.411229 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.411235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.412579 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.412582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.414335 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.414348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.415230 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.415233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.415360 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.415403 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:13:28.415557 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 03:13:28.415701 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":119,"thread_start_us":81,"threads_started":1}
I20260929 03:13:28.415866 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.409283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.20.174.67:45741)
0929 03:13:28.409660 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:28.409673 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:28.410401 (+ 728us) server_negotiation.cc:407] Connection header received
0929 03:13:28.410512 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:28.410517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:28.410739 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:28.410904 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:28.411374 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.412445 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.414490 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.415080 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.415209 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.415387 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:28.415594 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 03:13:28.415700 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":185,"threads_started":1}
I20260929 03:13:28.416077 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.409879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.20.174.66:45887)
0929 03:13:28.410355 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:28.410370 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:28.410394 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:13:28.410512 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:28.410517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:28.410739 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:28.410904 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:28.411318 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.412484 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.414275 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.415166 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.415279 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.415437 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:28.415808 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 03:13:28.415915 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":145,"threads_started":1}
I20260929 03:13:28.416071 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.409984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:35253 (local address 127.0.0.1:48962)
0929 03:13:28.411335 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:28.411349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:28.411448 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:28.411959 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:28.411963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:28.411981 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:28.412087 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.412093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.413236 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.413240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.414385 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.414395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.415648 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.415651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.415772 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.415793 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:13:28.415853 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:13:28.415917 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1255,"mutex_wait_us":8,"thread_start_us":87,"threads_started":1}
I20260929 03:13:28.416390 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.409138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:45887 (local address 127.0.0.1:52472)
0929 03:13:28.410214 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:28.410229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:28.410376 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:28.410917 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:28.410923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:28.410946 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:28.411107 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.411115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.412650 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.412653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.414166 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.414176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.415504 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.415507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.415607 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.415637 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:13:28.415708 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:13:28.416251 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":468,"thread_start_us":90,"threads_started":1}
I20260929 03:13:28.416850 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.410412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.20.174.65:35253)
0929 03:13:28.410759 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:28.410765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:28.411457 (+ 692us) server_negotiation.cc:407] Connection header received
0929 03:13:28.411550 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:28.411558 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:28.411774 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:28.411929 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:28.412208 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.413116 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.414570 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.415365 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.415470 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.415594 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:28.416545 (+ 951us) server_negotiation.cc:299] Negotiation successful
0929 03:13:28.416691 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":142,"threads_started":1}
I20260929 03:13:28.419852 4244 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9e218c023945f2a56f90b0ea55f5bf (DEFAULT_TABLE table=test-table [id=35d92c269ed948cd9f431e51cc2a7b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:28.419854 4512 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9e218c023945f2a56f90b0ea55f5bf (DEFAULT_TABLE table=test-table [id=35d92c269ed948cd9f431e51cc2a7b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:28.421082 4244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9e218c023945f2a56f90b0ea55f5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:28.421084 4512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9e218c023945f2a56f90b0ea55f5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:28.422613 4378 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9e218c023945f2a56f90b0ea55f5bf (DEFAULT_TABLE table=test-table [id=35d92c269ed948cd9f431e51cc2a7b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:28.423586 4378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9e218c023945f2a56f90b0ea55f5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:28.428951 4595 tablet_bootstrap.cc:492] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Bootstrap starting.
I20260929 03:13:28.429440 4596 tablet_bootstrap.cc:492] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb: Bootstrap starting.
I20260929 03:13:28.430343 4597 tablet_bootstrap.cc:492] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400: Bootstrap starting.
I20260929 03:13:28.432747 4596 tablet_bootstrap.cc:654] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:28.433281 4597 tablet_bootstrap.cc:654] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:28.434280 4597 log.cc:824] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:28.434288 4596 log.cc:824] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:28.436326 4595 tablet_bootstrap.cc:654] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:28.436753 4597 tablet_bootstrap.cc:492] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400: No bootstrap required, opened a new log
I20260929 03:13:28.436980 4597 ts_tablet_manager.cc:1397] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 03:13:28.437211 4595 log.cc:824] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:28.439523 4596 tablet_bootstrap.cc:492] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb: No bootstrap required, opened a new log
I20260929 03:13:28.439589 4595 tablet_bootstrap.cc:492] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: No bootstrap required, opened a new log
I20260929 03:13:28.439786 4595 ts_tablet_manager.cc:1397] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260929 03:13:28.439822 4596 ts_tablet_manager.cc:1397] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260929 03:13:28.441381 4597 raft_consensus.cc:348] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:28.441756 4597 raft_consensus.cc:374] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:28.441838 4597 raft_consensus.cc:729] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2861338795d74860b8f4ebda76b6a400, State: Initialized, Role: FOLLOWER
I20260929 03:13:28.442337 4597 consensus_queue.cc:260] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:28.443136 4444 heartbeater.cc:503] Master 127.20.174.126:37661 was elected leader, sending a full tablet report...
I20260929 03:13:28.443513 4597 ts_tablet_manager.cc:1428] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:13:28.443755 4596 raft_consensus.cc:348] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:28.444022 4596 raft_consensus.cc:374] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:28.444078 4596 raft_consensus.cc:729] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84bac3401714a35a403c5c6d177dacb, State: Initialized, Role: FOLLOWER
I20260929 03:13:28.444577 4596 consensus_queue.cc:260] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:28.445307 4578 heartbeater.cc:503] Master 127.20.174.126:37661 was elected leader, sending a full tablet report...
I20260929 03:13:28.445659 4596 ts_tablet_manager.cc:1428] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260929 03:13:28.445514 4595 raft_consensus.cc:348] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:28.445811 4595 raft_consensus.cc:374] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:28.445868 4595 raft_consensus.cc:729] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8b6b547d594a72b4f4bcf02e504f4d, State: Initialized, Role: FOLLOWER
I20260929 03:13:28.446343 4595 consensus_queue.cc:260] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:28.447384 4595 ts_tablet_manager.cc:1428] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 03:13:28.447818 4310 heartbeater.cc:503] Master 127.20.174.126:37661 was elected leader, sending a full tablet report...
I20260929 03:13:28.469697 4603 raft_consensus.cc:482] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:13:28.469978 4603 raft_consensus.cc:504] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:28.471211 4603 leader_election.cc:288] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741), 2861338795d74860b8f4ebda76b6a400 (127.20.174.66:45887)
I20260929 03:13:28.477401 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.471551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:45741 (local address 127.20.174.65:38003)
0929 03:13:28.472057 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:28.472070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:28.472154 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:28.472485 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:28.472488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:28.472531 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:28.472760 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.472768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.473901 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.473904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.475586 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.475601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.476717 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.476721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.476968 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.477003 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 03:13:28.477137 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 03:13:28.477220 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":387,"spinlock_wait_cycles":30336,"thread_start_us":141,"threads_started":1}
I20260929 03:13:28.477365 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.471709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:35743 (local address 127.20.174.66:45887)
0929 03:13:28.471853 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:28.471858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:28.472086 (+ 228us) server_negotiation.cc:407] Connection header received
0929 03:13:28.472293 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:28.472297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:28.472352 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:28.472453 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:28.472940 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.473746 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.475781 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.476560 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.476713 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.476773 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:28.477125 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 03:13:28.477219 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:13:28.477370 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.471693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:38003 (local address 127.20.174.67:45741)
0929 03:13:28.471835 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:28.471841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:28.472170 (+ 329us) server_negotiation.cc:407] Connection header received
0929 03:13:28.472254 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:28.472257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:28.472313 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:28.472426 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:28.472943 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.473749 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.475833 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.476560 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.476713 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.476773 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:28.477177 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 03:13:28.477259 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:13:28.477824 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.471551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:45887 (local address 127.20.174.65:35743)
0929 03:13:28.471972 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:28.471987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:28.472133 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:28.472485 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:28.472488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:28.472533 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:28.472768 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.472774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.473895 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.473900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.475586 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:28.475601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.476717 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.476721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:28.476897 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.476929 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:13:28.477566 (+ 637us) client_negotiation.cc:240] Negotiation successful
0929 03:13:28.477659 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":214,"spinlock_wait_cycles":7040,"thread_start_us":119,"threads_started":1}
I20260929 03:13:28.478193 4532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9e218c023945f2a56f90b0ea55f5bf" candidate_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84bac3401714a35a403c5c6d177dacb" is_pre_election: true
I20260929 03:13:28.478605 4532 raft_consensus.cc:2454] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd8b6b547d594a72b4f4bcf02e504f4d in term 0.
I20260929 03:13:28.478639 4398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9e218c023945f2a56f90b0ea55f5bf" candidate_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2861338795d74860b8f4ebda76b6a400" is_pre_election: true
I20260929 03:13:28.478920 4398 raft_consensus.cc:2454] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd8b6b547d594a72b4f4bcf02e504f4d in term 0.
I20260929 03:13:28.479208 4198 leader_election.cc:302] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a84bac3401714a35a403c5c6d177dacb, fd8b6b547d594a72b4f4bcf02e504f4d; no voters:
I20260929 03:13:28.479568 4603 raft_consensus.cc:2781] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:13:28.479691 4603 raft_consensus.cc:482] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:13:28.479861 4603 raft_consensus.cc:3037] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:28.481318 4603 raft_consensus.cc:504] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:28.481776 4603 leader_election.cc:288] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [CANDIDATE]: Term 1 election: Requested vote from peers a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741), 2861338795d74860b8f4ebda76b6a400 (127.20.174.66:45887)
I20260929 03:13:28.482187 4532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9e218c023945f2a56f90b0ea55f5bf" candidate_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84bac3401714a35a403c5c6d177dacb"
I20260929 03:13:28.482219 4398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9e218c023945f2a56f90b0ea55f5bf" candidate_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2861338795d74860b8f4ebda76b6a400"
I20260929 03:13:28.482312 4532 raft_consensus.cc:3037] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:28.482371 4398 raft_consensus.cc:3037] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:28.483506 4532 raft_consensus.cc:2454] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd8b6b547d594a72b4f4bcf02e504f4d in term 1.
I20260929 03:13:28.483716 4398 raft_consensus.cc:2454] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd8b6b547d594a72b4f4bcf02e504f4d in term 1.
I20260929 03:13:28.483964 4198 leader_election.cc:302] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a84bac3401714a35a403c5c6d177dacb, fd8b6b547d594a72b4f4bcf02e504f4d; no voters:
I20260929 03:13:28.484164 4603 raft_consensus.cc:2781] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:13:28.484419 4603 raft_consensus.cc:686] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 LEADER]: Becoming Leader. State: Replica: fd8b6b547d594a72b4f4bcf02e504f4d, State: Running, Role: LEADER
I20260929 03:13:28.484825 4603 consensus_queue.cc:237] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:28.488159 4119 catalog_manager.cc:5495] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d reported cstate change: term changed from 0 to 1, leader changed from <none> to fd8b6b547d594a72b4f4bcf02e504f4d (127.20.174.65). New cstate: current_term: 1 leader_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } health_report { overall_health: HEALTHY } } }
I20260929 03:13:28.506682 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.502819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.20.174.65:35253)
0929 03:13:28.503008 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:28.503012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:28.503026 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:13:28.503192 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:28.503195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:28.503251 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:28.503344 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:13:28.503575 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.504121 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.505121 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:28.505341 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:28.505475 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:28.505804 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:13:28.506251 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:13:28.506346 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:28.506492 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:13:28.506561 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:13:28.512965 4532 raft_consensus.cc:1261] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb [term 1 FOLLOWER]: Refusing update from remote peer fd8b6b547d594a72b4f4bcf02e504f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:28.512965 4398 raft_consensus.cc:1261] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Refusing update from remote peer fd8b6b547d594a72b4f4bcf02e504f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:28.513725 4603 consensus_queue.cc:1035] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:13:28.513984 4606 consensus_queue.cc:1035] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:13:28.533264 4119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:13:28.533497 4119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:13:28.536352 4119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Sending DeleteTablet for 3 replicas of tablet 6c9e218c023945f2a56f90b0ea55f5bf
I20260929 03:13:28.537358 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 4181
I20260929 03:13:28.537415 4512 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9e218c023945f2a56f90b0ea55f5bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:28 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:36500
I20260929 03:13:28.537608 4378 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9e218c023945f2a56f90b0ea55f5bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:28 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:52472
I20260929 03:13:28.539070 4619 tablet_replica.cc:330] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb: stopping tablet replica
I20260929 03:13:28.539281 4620 tablet_replica.cc:330] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400: stopping tablet replica
I20260929 03:13:28.539498 4619 raft_consensus.cc:2229] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:13:28.539672 4620 raft_consensus.cc:2229] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:13:28.539925 4619 raft_consensus.cc:2258] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:28.540023 4620 raft_consensus.cc:2258] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:28.541488 4619 ts_tablet_manager.cc:1905] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:13:28.541596 4620 ts_tablet_manager.cc:1905] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:13:28.544322 4619 ts_tablet_manager.cc:1918] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:13:28.544450 4619 log.cc:1196] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/6c9e218c023945f2a56f90b0ea55f5bf
I20260929 03:13:28.544883 4619 ts_tablet_manager.cc:1939] T 6c9e218c023945f2a56f90b0ea55f5bf P a84bac3401714a35a403c5c6d177dacb: Deleting consensus metadata
I20260929 03:13:28.545100 4620 ts_tablet_manager.cc:1918] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:13:28.545199 4620 log.cc:1196] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/6c9e218c023945f2a56f90b0ea55f5bf
I20260929 03:13:28.545573 4620 ts_tablet_manager.cc:1939] T 6c9e218c023945f2a56f90b0ea55f5bf P 2861338795d74860b8f4ebda76b6a400: Deleting consensus metadata
W20260929 03:13:28.545590 4105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:35253 (error 108)
I20260929 03:13:28.545831 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 4315
I20260929 03:13:28.547415 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.547124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:35253 (local address 127.0.0.1:48988)
0929 03:13:28.547267 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:28.547339 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 03:13:28.547691 4104 catalog_manager.cc:4841] TS a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741): tablet 6c9e218c023945f2a56f90b0ea55f5bf (table test-table [id=35d92c269ed948cd9f431e51cc2a7b42]) successfully deleted
W20260929 03:13:28.547940 4105 catalog_manager.cc:4584] TS fd8b6b547d594a72b4f4bcf02e504f4d (127.20.174.65:35253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c9e218c023945f2a56f90b0ea55f5bf: Network error: Client connection negotiation failed: client connection to 127.20.174.65:35253: connect: Connection refused (error 111)
I20260929 03:13:28.551694 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 4449
I20260929 03:13:28.553072 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:28.551808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:45887 (local address 127.0.0.1:52482)
0929 03:13:28.552896 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:28.552967 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.66:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":976}
I20260929 03:13:28.559067 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 4087
I20260929 03:13:48.565768 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:37661
--webserver_interface=127.20.174.126
--webserver_port=45619
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:37661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:13:48.684703 4622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:48.685024 4622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:48.685066 4622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:48.689436 4622 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:13:48.689525 4622 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:13:48.689543 4622 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:13:48.689556 4622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:13:48.689575 4622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:48.689594 4622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:48.689611 4622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:13:48.689662 4622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:13:48.695510 4622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:37661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:37661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=45619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:48.696766 4622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:48.697894 4622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:48.712857 4628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:48.713045 4622 server_base.cc:1034] running on GCE node
W20260929 03:13:48.712859 4631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:48.712859 4629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:48.714035 4622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:48.715454 4622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:48.716660 4622 hybrid_clock.cc:648] HybridClock initialized: now 1790651628716592 us; error 85 us; skew 500 ppm
Sep 29 03:13:48 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651628, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:48.719977 4622 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:13:48.721026 4622 webserver.cc:460] Webserver started at http://127.20.174.126:45619/ using document root <none> and password file <none>
I20260929 03:13:48.721676 4622 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:48.721809 4622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:48.726739 4622 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:48.729393 4638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:48.731122 4622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:48.731298 4622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21"
format_stamp: "Formatted at 2026-09-29 03:13:27 on dist-test-slave-k1dg"
server_key: "bdb9e94d2387baced4903a0be9c349f5"
server_key_iv: "6b4f1a11691ebf8a8b001302e2650a4c"
server_key_version: "encryptionkey@0"
I20260929 03:13:48.732043 4622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:48.749186 4622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:48.753676 4622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:48.753917 4622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:48.761494 4622 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:37661
I20260929 03:13:48.761791 4690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:37661 every 8 connection(s)
I20260929 03:13:48.762867 4622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:13:48.763859 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 4622
I20260929 03:13:48.764381 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:35253
--local_ip_for_outbound_sockets=127.20.174.65
--tserver_master_addrs=127.20.174.126:37661
--webserver_port=34011
--webserver_interface=127.20.174.65
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:13:48.768889 4691 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:13:48.770803 4691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Bootstrap starting.
I20260929 03:13:48.789330 4691 log.cc:824] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:48.798771 4691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:13:48.799218 4691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Bootstrap complete.
I20260929 03:13:48.803517 4691 raft_consensus.cc:348] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:13:48.803996 4691 raft_consensus.cc:729] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d389bb2c24cb4471b8ffbeaa0bc93b21, State: Initialized, Role: FOLLOWER
I20260929 03:13:48.804549 4691 consensus_queue.cc:260] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:13:48.804697 4691 raft_consensus.cc:388] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:13:48.804791 4691 raft_consensus.cc:482] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:13:48.804908 4691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:13:48.807060 4691 raft_consensus.cc:504] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:13:48.807433 4691 leader_election.cc:302] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d389bb2c24cb4471b8ffbeaa0bc93b21; no voters:
I20260929 03:13:48.807814 4691 leader_election.cc:288] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:13:48.807914 4695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:13:48.808130 4695 raft_consensus.cc:686] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 2 LEADER]: Becoming Leader. State: Replica: d389bb2c24cb4471b8ffbeaa0bc93b21, State: Running, Role: LEADER
I20260929 03:13:48.808455 4695 consensus_queue.cc:237] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:13:48.809144 4691 sys_catalog.cc:564] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:13:48.809487 4695 sys_catalog.cc:455] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d389bb2c24cb4471b8ffbeaa0bc93b21. Latest consensus state: current_term: 2 leader_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } } }
I20260929 03:13:48.809619 4695 sys_catalog.cc:458] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:48.810006 4699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:13:48.810029 4696 sys_catalog.cc:455] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } } }
I20260929 03:13:48.810271 4696 sys_catalog.cc:458] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: This master's current role is: LEADER
I20260929 03:13:48.815440 4699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:13:48.816321 4699 catalog_manager.cc:1259] Loaded cluster ID: 46e16f05f81a41329d44ce3316cf4ee0
I20260929 03:13:48.816390 4699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:13:48.819119 4699 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:13:48.819978 4699 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Loaded TSK: 0
I20260929 03:13:48.834807 4699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Generated new TSK 1
I20260929 03:13:48.835139 4699 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:13:48.893083 4693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:48.893427 4693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:48.893510 4693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:48.897852 4693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:48.897972 4693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:48.898073 4693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:13:48.904027 4693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:35253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=34011
--tserver_master_addrs=127.20.174.126:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:48.905170 4693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:48.906262 4693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:13:48.918843 4693 server_base.cc:1034] running on GCE node
W20260929 03:13:48.919028 4718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:48.918848 4721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:48.919204 4719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:48.919759 4693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:48.920588 4693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:48.921826 4693 hybrid_clock.cc:648] HybridClock initialized: now 1790651628921788 us; error 61 us; skew 500 ppm
Sep 29 03:13:48 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651628, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:48.925181 4693 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:13:48.926237 4693 webserver.cc:460] Webserver started at http://127.20.174.65:34011/ using document root <none> and password file <none>
I20260929 03:13:48.926955 4693 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:48.927078 4693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:48.931568 4693 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:13:48.934307 4728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:48.936177 4693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:13:48.936352 4693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "fd8b6b547d594a72b4f4bcf02e504f4d"
format_stamp: "Formatted at 2026-09-29 03:13:27 on dist-test-slave-k1dg"
server_key: "079d5834a2dc57ac5d60d33145977fd8"
server_key_iv: "091e7f76fa6625b788b491cf7bdcccb7"
server_key_version: "encryptionkey@0"
I20260929 03:13:48.937077 4693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:48.947408 4693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:48.951110 4693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:48.951331 4693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:48.951987 4693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:48.953191 4735 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:13:48.957913 4693 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:13:48.957990 4693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 03:13:48.958066 4693 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:13:48.959918 4693 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:13:48.959983 4693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 03:13:48.960259 4735 tablet_bootstrap.cc:492] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Bootstrap starting.
I20260929 03:13:48.971547 4693 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:35253
I20260929 03:13:48.971853 4842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:35253 every 8 connection(s)
I20260929 03:13:48.972955 4693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:13:48.981626 4735 log.cc:824] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:48.982415 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 4693
I20260929 03:13:48.982976 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:45887
--local_ip_for_outbound_sockets=127.20.174.66
--tserver_master_addrs=127.20.174.126:37661
--webserver_port=42225
--webserver_interface=127.20.174.66
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:13:48 dist-test-slave-k1dg krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651628, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:48.990861 4735 tablet_bootstrap.cc:492] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:13:48.991361 4735 tablet_bootstrap.cc:492] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Bootstrap complete.
I20260929 03:13:48.992718 4735 ts_tablet_manager.cc:1397] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.009s
I20260929 03:13:48.994099 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:48.975426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:54131)
0929 03:13:48.976278 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:48.976342 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:48.977235 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:48.978553 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:48.978562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:48.979010 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:48.979493 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:48.979508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:48.980875 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:48.980879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:48.981874 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:48.981887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:48.983004 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:48.983009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:48.983299 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:48.984520 (+ 1221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:48.984542 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:48.986569 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:48.989934 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:48.989941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:48.989954 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:48.990317 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:48.990968 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:48.990971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:48.990973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:48.991163 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:48.992061 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:48.992071 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:48.992395 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 03:13:48.992616 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 03:13:48.993080 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":188,"threads_started":1}
I20260929 03:13:48.994252 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:48.975415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:54131 (local address 127.20.174.126:37661)
0929 03:13:48.975911 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:48.975934 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:48.976497 (+ 563us) server_negotiation.cc:407] Connection header received
0929 03:13:48.977706 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:48.977717 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:48.978050 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:48.978412 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:48.979709 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:48.980719 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:48.982016 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:48.982844 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:48.983032 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:48.986929 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:48.986939 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:48.986961 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:48.987006 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:48.989753 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:48.990506 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:48.990516 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:48.990525 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:48.990831 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:48.991303 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:48.991309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:48.991314 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:48.991922 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:48.992088 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:48.992800 (+ 712us) server_negotiation.cc:299] Negotiation successful
0929 03:13:48.993181 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":115,"threads_started":1}
I20260929 03:13:48.996152 4735 raft_consensus.cc:348] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:48.996723 4735 raft_consensus.cc:729] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8b6b547d594a72b4f4bcf02e504f4d, State: Initialized, Role: FOLLOWER
I20260929 03:13:48.997263 4735 consensus_queue.cc:260] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:48.998112 4843 heartbeater.cc:347] Connected to a master server at 127.20.174.126:37661
I20260929 03:13:48.998371 4843 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:48.998467 4735 ts_tablet_manager.cc:1428] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:13:48.999053 4843 heartbeater.cc:511] Master 127.20.174.126:37661 requested a full tablet report, sending...
I20260929 03:13:49.002177 4655 ts_manager.cc:194] Registered new tserver with Master: fd8b6b547d594a72b4f4bcf02e504f4d (127.20.174.65:35253)
I20260929 03:13:49.006805 4655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:54131
I20260929 03:13:49.010031 4843 heartbeater.cc:503] Master 127.20.174.126:37661 was elected leader, sending a full tablet report...
Sep 29 03:13:49 dist-test-slave-k1dg krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651628, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:13:49.019064 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.006191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:35253 (local address 127.0.0.1:55818)
0929 03:13:49.007205 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:49.007246 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:49.007443 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:49.008129 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:49.008135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:49.008520 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:49.008635 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.008643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.009952 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.009956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.010982 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.010993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.011882 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.011886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.012010 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.012644 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:49.012656 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:49.014105 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:49.016614 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:49.016625 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:49.016631 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:49.016974 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:49.017362 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:49.017365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:49.017366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:49.017476 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:49.018263 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:49.018271 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:49.018423 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 03:13:49.018620 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 03:13:49.018902 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":129,"threads_started":1}
I20260929 03:13:49.019119 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.006493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.20.174.65:35253)
0929 03:13:49.006926 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:49.006938 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:49.007370 (+ 432us) server_negotiation.cc:407] Connection header received
0929 03:13:49.007608 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:49.007614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:49.007895 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:49.008052 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:49.008778 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.009824 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.011120 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.011652 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.011774 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.014273 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:49.014286 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:49.014300 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:49.014329 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:49.016447 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:49.017107 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:49.017113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:49.017115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:49.017223 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:49.017577 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:49.017580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:49.017582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:49.018050 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:49.018196 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:49.018754 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 03:13:49.018949 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":105,"threads_started":1}
I20260929 03:13:49.020476 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9e218c023945f2a56f90b0ea55f5bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:28 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:55818
I20260929 03:13:49.021517 4852 tablet_replica.cc:330] stopping tablet replica
I20260929 03:13:49.021857 4852 raft_consensus.cc:2229] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:13:49.022135 4852 raft_consensus.cc:2258] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:49.023221 4852 ts_tablet_manager.cc:1905] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:13:49.027344 4852 ts_tablet_manager.cc:1918] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:13:49.027503 4852 log.cc:1196] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal/wals/6c9e218c023945f2a56f90b0ea55f5bf
I20260929 03:13:49.027948 4852 ts_tablet_manager.cc:1939] T 6c9e218c023945f2a56f90b0ea55f5bf P fd8b6b547d594a72b4f4bcf02e504f4d: Deleting consensus metadata
I20260929 03:13:49.030120 4641 catalog_manager.cc:4841] TS fd8b6b547d594a72b4f4bcf02e504f4d (127.20.174.65:35253): tablet 6c9e218c023945f2a56f90b0ea55f5bf (table test-table [id=35d92c269ed948cd9f431e51cc2a7b42]) successfully deleted
W20260929 03:13:49.113155 4848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:49.113554 4848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:49.113655 4848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:49.118078 4848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:49.118198 4848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:49.118296 4848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:13:49.124114 4848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:45887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=42225
--tserver_master_addrs=127.20.174.126:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:49.125363 4848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:49.126587 4848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:49.137089 4860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:49.137101 4857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:49.137101 4858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:49.137348 4848 server_base.cc:1034] running on GCE node
I20260929 03:13:49.137885 4848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:49.138691 4848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:49.139937 4848 hybrid_clock.cc:648] HybridClock initialized: now 1790651629139922 us; error 49 us; skew 500 ppm
Sep 29 03:13:49 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651629, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:49.143281 4848 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:13:49.144263 4848 webserver.cc:460] Webserver started at http://127.20.174.66:42225/ using document root <none> and password file <none>
I20260929 03:13:49.144831 4848 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:49.144912 4848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:49.149106 4848 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:13:49.151835 4867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:49.153398 4848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:49.153514 4848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "2861338795d74860b8f4ebda76b6a400"
format_stamp: "Formatted at 2026-09-29 03:13:28 on dist-test-slave-k1dg"
server_key: "95f5ca700006673fd7d96e9a8e377414"
server_key_iv: "7b4cde3c8f90618d3f8bf02ab50fa678"
server_key_version: "encryptionkey@0"
I20260929 03:13:49.154083 4848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:49.168641 4848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:49.172343 4848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:49.172518 4848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:49.173097 4848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:49.173980 4848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:13:49.174034 4848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:49.174079 4848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:13:49.174095 4848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:49.183854 4848 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:45887
I20260929 03:13:49.183959 4980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:45887 every 8 connection(s)
I20260929 03:13:49.184966 4848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:13:49.191673 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 4848
I20260929 03:13:49.192209 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:45741
--local_ip_for_outbound_sockets=127.20.174.67
--tserver_master_addrs=127.20.174.126:37661
--webserver_port=35377
--webserver_interface=127.20.174.67
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:13:49 dist-test-slave-k1dg krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651629, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:49.200938 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.186996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:37201 (local address 127.20.174.126:37661)
0929 03:13:49.187147 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:49.187151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:49.188144 (+ 993us) server_negotiation.cc:407] Connection header received
0929 03:13:49.189511 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:49.189514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:49.189594 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:49.189737 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:49.191362 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.192169 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.193881 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.194128 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.194241 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.196868 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:49.196872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:49.196874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:49.196909 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:49.198689 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:49.199230 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:49.199234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:49.199236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:49.199327 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:49.199739 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:49.199742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:49.199743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:49.199936 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:49.200043 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:49.200590 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 03:13:49.200751 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:13:49.201871 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.187352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:37201)
0929 03:13:49.187965 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:49.188011 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:49.189284 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:49.189990 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:49.189998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:49.190527 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:49.191190 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.191207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.192491 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.192494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.193743 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.193755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.194307 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.194312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.194554 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.195190 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:49.195207 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:49.196682 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:49.198810 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:49.198816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:49.198834 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:49.199112 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:49.199420 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:49.199424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:49.199426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:49.199575 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:49.200054 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:49.200063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:49.200336 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 03:13:49.200546 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 03:13:49.200895 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":153,"threads_started":1}
I20260929 03:13:49.203227 4981 heartbeater.cc:347] Connected to a master server at 127.20.174.126:37661
I20260929 03:13:49.203518 4981 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:49.204114 4981 heartbeater.cc:511] Master 127.20.174.126:37661 requested a full tablet report, sending...
I20260929 03:13:49.205389 4655 ts_manager.cc:194] Registered new tserver with Master: 2861338795d74860b8f4ebda76b6a400 (127.20.174.66:45887)
I20260929 03:13:49.206023 4655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:37201
W20260929 03:13:49.313357 4984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:13:49.313634 4984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:13:49.313721 4984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:13:49.318001 4984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:13:49.318073 4984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:13:49.318176 4984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:13:49.323844 4984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:45741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=35377
--tserver_master_addrs=127.20.174.126:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:13:49.325017 4984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:13:49.326120 4984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:13:49.336869 4990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:49.336987 4992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:13:49.336871 4989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:13:49.337555 4984 server_base.cc:1034] running on GCE node
I20260929 03:13:49.337992 4984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:13:49.338819 4984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:13:49.340030 4984 hybrid_clock.cc:648] HybridClock initialized: now 1790651629340018 us; error 28 us; skew 500 ppm
Sep 29 03:13:49 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651629, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:13:49.343168 4984 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:13:49.344157 4984 webserver.cc:460] Webserver started at http://127.20.174.67:35377/ using document root <none> and password file <none>
I20260929 03:13:49.344784 4984 fs_manager.cc:359] Metadata directory not provided
I20260929 03:13:49.344863 4984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:13:49.349093 4984 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:13:49.351651 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:49.353119 4984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:13:49.353255 4984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "a84bac3401714a35a403c5c6d177dacb"
format_stamp: "Formatted at 2026-09-29 03:13:28 on dist-test-slave-k1dg"
server_key: "a680bd30d5dd4422e86c7486c0130ece"
server_key_iv: "a5579bc8bd436b17370d3cfbd329bde4"
server_key_version: "encryptionkey@0"
I20260929 03:13:49.353976 4984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:13:49.391338 4984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:13:49.395084 4984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:13:49.395334 4984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:13:49.396034 4984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:13:49.396994 4984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:13:49.397045 4984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:49.397094 4984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:13:49.397153 4984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:13:49.406682 4984 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:45741
I20260929 03:13:49.406699 5112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:45741 every 8 connection(s)
I20260929 03:13:49.407886 4984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:13:49.410079 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 4984
Sep 29 03:13:49 dist-test-slave-k1dg krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651629, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:13:49.420332 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.410938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.20.174.126:37661)
0929 03:13:49.411161 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:49.411166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:49.411381 (+ 215us) server_negotiation.cc:407] Connection header received
0929 03:13:49.411504 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:49.411507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:49.411561 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:49.411659 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:49.412429 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.413247 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.414138 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.414519 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.414610 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.416175 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:49.416179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:49.416182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:49.416209 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:49.418223 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:49.418765 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:49.418769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:49.418771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:49.418873 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:49.419270 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:49.419274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:49.419276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:49.419470 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:49.419760 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:49.419970 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 03:13:49.420123 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260929 03:13:49.424615 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.409998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:55975 (local address 127.20.174.126:37661)
0929 03:13:49.410154 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:49.410178 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:49.411408 (+ 1230us) server_negotiation.cc:407] Connection header received
0929 03:13:49.413046 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:49.413049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:49.413092 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:49.413158 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:49.414560 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.415354 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.416949 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.417215 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.417335 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.420469 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:49.420475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:49.420479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:49.420513 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:49.422189 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:49.422873 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:49.422882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:49.422883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:49.422958 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:49.423369 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:49.423372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:49.423374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:49.423556 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:49.423661 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:49.424221 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 03:13:49.424443 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:13:49.425429 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.410566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:55975)
0929 03:13:49.411204 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:49.411261 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:49.412818 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:49.413367 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:49.413376 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:49.413759 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:13:49.414349 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.414369 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.415508 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.415511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.416795 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.416806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.417401 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.417407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.417645 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.418194 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:13:49.418217 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:13:49.420257 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:13:49.422331 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:49.422338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:49.422355 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:49.422763 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:49.423057 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:13:49.423060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:13:49.423062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:13:49.423260 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:13:49.423673 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:13:49.423681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:13:49.423953 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 03:13:49.424166 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 03:13:49.424541 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":182,"threads_started":1}
I20260929 03:13:49.426645 5113 heartbeater.cc:347] Connected to a master server at 127.20.174.126:37661
I20260929 03:13:49.426921 5113 heartbeater.cc:464] Registering TS with master...
I20260929 03:13:49.427489 5113 heartbeater.cc:511] Master 127.20.174.126:37661 requested a full tablet report, sending...
I20260929 03:13:49.428629 4655 ts_manager.cc:194] Registered new tserver with Master: a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741)
I20260929 03:13:49.429271 4655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:55975
I20260929 03:13:49.434595 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:13:49.438807 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.435360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.20.174.126:37661)
0929 03:13:49.435516 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:49.435519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:49.435652 (+ 133us) server_negotiation.cc:407] Connection header received
0929 03:13:49.435824 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:49.435826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:49.435927 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:49.436014 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:13:49.436268 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.436849 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.437693 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.437936 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.438013 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.438287 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:13:49.438480 (+ 193us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 03:13:49.438697 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44642: token expired
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 03:13:49.438898 4845 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44642: token expired
W20260929 03:13:49.439297 4584 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.174.126:37661)
I20260929 03:13:49.447065 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.439736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.20.174.126:37661)
0929 03:13:49.439869 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:49.439873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:49.439891 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:13:49.439922 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:49.439924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:49.439958 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:49.440027 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:13:49.440650 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.441223 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.442118 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.442417 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.442559 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.443848 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:13:49.443856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:13:49.443858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:13:49.443883 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:13:49.445363 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:49.445842 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:49.445845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:49.445846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:49.445941 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:13:49.446201 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:13:49.446204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:13:49.446205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:13:49.446386 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:13:49.446577 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:49.446824 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 03:13:49.446935 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:13:49.450125 4655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:13:49.453665 4655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:13:49.461951 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 123340c510f0495481c7413bfec2760d (DEFAULT_TABLE table=test-table [id=cf9107f902834222b1643e9fc40272da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:49.462733 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123340c510f0495481c7413bfec2760d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:49.465508 4735 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: Bootstrap starting.
I20260929 03:13:49.466507 4735 tablet_bootstrap.cc:654] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:49.467322 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.461348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:45741 (local address 127.0.0.1:42330)
0929 03:13:49.461874 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:49.461888 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:49.461983 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:49.462668 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:49.462671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:49.462690 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:49.462929 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.462939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.464295 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.464299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.465485 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.465495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.466790 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.466794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.466903 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.466966 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 03:13:49.467037 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:13:49.467122 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":113,"threads_started":1}
I20260929 03:13:49.467517 5124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.461767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.20.174.67:45741)
0929 03:13:49.462161 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:49.462173 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:49.462194 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:13:49.462271 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:49.462279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:49.462514 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:49.462687 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:49.463063 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.464156 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.465733 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.466666 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.466918 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.467100 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:49.467167 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 03:13:49.467266 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":98,"threads_started":1}
I20260929 03:13:49.467679 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.461020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:45887 (local address 127.0.0.1:54994)
0929 03:13:49.461138 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:49.461154 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:49.461278 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:49.462391 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:49.462394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:49.462412 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:49.462581 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.462592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.464289 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.464293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.465241 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.465252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.467261 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.467264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.467367 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.467399 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:13:49.467457 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:13:49.467529 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":59}
I20260929 03:13:49.468254 4735 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: No bootstrap required, opened a new log
I20260929 03:13:49.468349 4735 ts_tablet_manager.cc:1397] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 03:13:49.468504 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.461388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.20.174.66:45887)
0929 03:13:49.461811 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:49.461826 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:49.461855 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:13:49.461944 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:49.461950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:49.462198 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:49.462725 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:49.462913 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.464126 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.465380 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.467126 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.467953 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.468150 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:49.468205 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:13:49.468339 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":189,"threads_started":1}
I20260929 03:13:49.468815 4735 raft_consensus.cc:348] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:49.468942 4735 raft_consensus.cc:374] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:49.468988 4735 raft_consensus.cc:729] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8b6b547d594a72b4f4bcf02e504f4d, State: Initialized, Role: FOLLOWER
I20260929 03:13:49.469110 4735 consensus_queue.cc:260] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:49.469424 4735 ts_tablet_manager.cc:1428] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:13:49.471252 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 123340c510f0495481c7413bfec2760d (DEFAULT_TABLE table=test-table [id=cf9107f902834222b1643e9fc40272da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:49.471237 5047 tablet_service.cc:1467] Processing CreateTablet for tablet 123340c510f0495481c7413bfec2760d (DEFAULT_TABLE table=test-table [id=cf9107f902834222b1643e9fc40272da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:13:49.472303 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123340c510f0495481c7413bfec2760d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:13:49.472312 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123340c510f0495481c7413bfec2760d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 03:13:49.474637 4844 tablet.cc:2369] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:13:49.479554 5126 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Bootstrap starting.
I20260929 03:13:49.479553 5127 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Bootstrap starting.
I20260929 03:13:49.482092 5127 tablet_bootstrap.cc:654] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:49.482180 5126 tablet_bootstrap.cc:654] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Neither blocks nor log segments found. Creating new log.
I20260929 03:13:49.483104 5127 log.cc:824] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:49.483104 5126 log.cc:824] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Log is configured to *not* fsync() on all Append() calls
I20260929 03:13:49.485057 5126 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: No bootstrap required, opened a new log
I20260929 03:13:49.485064 5127 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: No bootstrap required, opened a new log
I20260929 03:13:49.485265 5126 ts_tablet_manager.cc:1397] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 03:13:49.485280 5127 ts_tablet_manager.cc:1397] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:13:49.488397 5126 raft_consensus.cc:348] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:49.488654 5126 raft_consensus.cc:374] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:49.488708 5126 raft_consensus.cc:729] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2861338795d74860b8f4ebda76b6a400, State: Initialized, Role: FOLLOWER
I20260929 03:13:49.488585 5127 raft_consensus.cc:348] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:49.488880 5127 raft_consensus.cc:374] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:13:49.488942 5127 raft_consensus.cc:729] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84bac3401714a35a403c5c6d177dacb, State: Initialized, Role: FOLLOWER
I20260929 03:13:49.489241 5126 consensus_queue.cc:260] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:49.489579 5127 consensus_queue.cc:260] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:49.490015 4981 heartbeater.cc:503] Master 127.20.174.126:37661 was elected leader, sending a full tablet report...
I20260929 03:13:49.490278 5126 ts_tablet_manager.cc:1428] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 03:13:49.490577 5127 ts_tablet_manager.cc:1428] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:13:49.490836 5113 heartbeater.cc:503] Master 127.20.174.126:37661 was elected leader, sending a full tablet report...
W20260929 03:13:49.658981 5114 tablet.cc:2369] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:13:49.686180 4982 tablet.cc:2369] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:13:49.862728 5130 raft_consensus.cc:482] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:13:49.862989 5130 raft_consensus.cc:504] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:49.864157 5130 leader_election.cc:288] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741), fd8b6b547d594a72b4f4bcf02e504f4d (127.20.174.65:35253)
I20260929 03:13:49.868155 5124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.864550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:46635 (local address 127.20.174.67:45741)
0929 03:13:49.864791 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:49.864798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:49.864891 (+ 93us) server_negotiation.cc:407] Connection header received
0929 03:13:49.865080 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:49.865082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:49.865123 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:49.865207 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:49.865480 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.865993 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.867160 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.867564 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.867712 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.867768 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:49.867955 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 03:13:49.868027 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260929 03:13:49.868171 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.864444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:45741 (local address 127.20.174.66:46635)
0929 03:13:49.864786 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:49.864802 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:49.864961 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:49.865219 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:49.865222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:49.865238 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:49.865366 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.865371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.866109 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.866114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.867020 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.867031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.867713 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.867717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.867818 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.867845 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 03:13:49.867939 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 03:13:49.868013 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":77,"thread_start_us":70,"threads_started":1}
I20260929 03:13:49.868299 5133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.864444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:35253 (local address 127.20.174.66:53589)
0929 03:13:49.864863 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:49.864882 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:13:49.865022 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:13:49.865373 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:13:49.865376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:13:49.865393 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:13:49.865517 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.865522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.866249 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.866254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.867122 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:13:49.867131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.867836 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.867840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:13:49.867942 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.867963 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:13:49.868028 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:13:49.868100 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260929 03:13:49.868502 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.864540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:53589 (local address 127.20.174.65:35253)
0929 03:13:49.864825 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:49.864830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:49.864974 (+ 144us) server_negotiation.cc:407] Connection header received
0929 03:13:49.865130 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:49.865133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:49.865191 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:49.865365 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:13:49.865626 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.866135 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.867254 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.867709 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.867849 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.868019 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:49.868057 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 03:13:49.868129 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260929 03:13:49.868952 5067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123340c510f0495481c7413bfec2760d" candidate_uuid: "2861338795d74860b8f4ebda76b6a400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84bac3401714a35a403c5c6d177dacb" is_pre_election: true
I20260929 03:13:49.869151 4797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123340c510f0495481c7413bfec2760d" candidate_uuid: "2861338795d74860b8f4ebda76b6a400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" is_pre_election: true
I20260929 03:13:49.869309 5067 raft_consensus.cc:2454] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2861338795d74860b8f4ebda76b6a400 in term 0.
I20260929 03:13:49.869483 4797 raft_consensus.cc:2454] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2861338795d74860b8f4ebda76b6a400 in term 0.
I20260929 03:13:49.869839 4869 leader_election.cc:302] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2861338795d74860b8f4ebda76b6a400, a84bac3401714a35a403c5c6d177dacb; no voters:
I20260929 03:13:49.870113 5130 raft_consensus.cc:2781] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:13:49.870193 5130 raft_consensus.cc:482] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:13:49.870235 5130 raft_consensus.cc:3037] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:49.871307 5130 raft_consensus.cc:504] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:49.871667 5130 leader_election.cc:288] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [CANDIDATE]: Term 1 election: Requested vote from peers a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741), fd8b6b547d594a72b4f4bcf02e504f4d (127.20.174.65:35253)
I20260929 03:13:49.872051 5067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123340c510f0495481c7413bfec2760d" candidate_uuid: "2861338795d74860b8f4ebda76b6a400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84bac3401714a35a403c5c6d177dacb"
I20260929 03:13:49.872085 4797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123340c510f0495481c7413bfec2760d" candidate_uuid: "2861338795d74860b8f4ebda76b6a400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d"
I20260929 03:13:49.872197 4797 raft_consensus.cc:3037] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:49.872197 5067 raft_consensus.cc:3037] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:13:49.873487 4797 raft_consensus.cc:2454] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2861338795d74860b8f4ebda76b6a400 in term 1.
I20260929 03:13:49.873553 5067 raft_consensus.cc:2454] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2861338795d74860b8f4ebda76b6a400 in term 1.
I20260929 03:13:49.873849 4870 leader_election.cc:302] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2861338795d74860b8f4ebda76b6a400, fd8b6b547d594a72b4f4bcf02e504f4d; no voters:
I20260929 03:13:49.874047 5130 raft_consensus.cc:2781] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:13:49.874228 5130 raft_consensus.cc:686] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 1 LEADER]: Becoming Leader. State: Replica: 2861338795d74860b8f4ebda76b6a400, State: Running, Role: LEADER
I20260929 03:13:49.874589 5130 consensus_queue.cc:237] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:13:49.878032 4654 catalog_manager.cc:5495] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2861338795d74860b8f4ebda76b6a400 (127.20.174.66). New cstate: current_term: 1 leader_uuid: "2861338795d74860b8f4ebda76b6a400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } health_report { overall_health: UNKNOWN } } }
I20260929 03:13:49.927099 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.923450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.20.174.66:45887)
0929 03:13:49.923581 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:13:49.923585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:13:49.923619 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:13:49.923702 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:13:49.923704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:13:49.923763 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:13:49.923870 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:13:49.924121 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.924657 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.925575 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:13:49.925789 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:13:49.925915 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:13:49.926160 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:13:49.926666 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:13:49.926762 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:13:49.926860 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 03:13:49.926920 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:13:49.933333 5067 raft_consensus.cc:1261] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 1 FOLLOWER]: Refusing update from remote peer 2861338795d74860b8f4ebda76b6a400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:49.933372 4797 raft_consensus.cc:1261] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Refusing update from remote peer 2861338795d74860b8f4ebda76b6a400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:13:49.934217 5130 consensus_queue.cc:1035] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:13:49.934515 5135 consensus_queue.cc:1035] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:13:49.942224 5139 mvcc.cc:205] Tried to move back new op lower bound from 7334509076199223296 to 7334509075969196032. Current Snapshot: MvccSnapshot[applied={T|T < 7334509076199223296}]
I20260929 03:13:49.942601 5138 mvcc.cc:205] Tried to move back new op lower bound from 7334509076199223296 to 7334509075969196032. Current Snapshot: MvccSnapshot[applied={T|T < 7334509076199223296}]
I20260929 03:13:49.945184 5140 mvcc.cc:205] Tried to move back new op lower bound from 7334509076199223296 to 7334509075969196032. Current Snapshot: MvccSnapshot[applied={T|T < 7334509076199223296}]
I20260929 03:13:49.951720 4654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:13:49.951905 4654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:13:49.954671 4654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Sending DeleteTablet for 3 replicas of tablet 123340c510f0495481c7413bfec2760d
I20260929 03:13:49.955591 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet 123340c510f0495481c7413bfec2760d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:49 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:55818
I20260929 03:13:49.955689 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 4693
I20260929 03:13:49.955633 5047 tablet_service.cc:1514] Processing DeleteTablet for tablet 123340c510f0495481c7413bfec2760d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:49 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:42330
I20260929 03:13:49.955852 4915 tablet_service.cc:1514] Processing DeleteTablet for tablet 123340c510f0495481c7413bfec2760d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:13:49 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:54994
I20260929 03:13:49.957857 5148 tablet_replica.cc:330] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: stopping tablet replica
I20260929 03:13:49.958323 5148 raft_consensus.cc:2229] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:13:49.958755 5148 raft_consensus.cc:2258] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:13:49.960448 5148 ts_tablet_manager.cc:1905] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:13:49.962249 4641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:35253 (error 108)
W20260929 03:13:49.963392 5122 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:13:49.963032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:35253 (local address 127.0.0.1:55820)
0929 03:13:49.963171 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:49.963288 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:35253: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 03:13:49.963446 5148 ts_tablet_manager.cc:1918] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:13:49.963548 5148 log.cc:1196] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/123340c510f0495481c7413bfec2760d
I20260929 03:13:49.963565 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 4848
W20260929 03:13:49.963593 4641 catalog_manager.cc:4584] TS fd8b6b547d594a72b4f4bcf02e504f4d (127.20.174.65:35253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 123340c510f0495481c7413bfec2760d: Network error: Client connection negotiation failed: client connection to 127.20.174.65:35253: connect: Connection reset by peer (error 104)
I20260929 03:13:49.964421 5148 ts_tablet_manager.cc:1939] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Deleting consensus metadata
I20260929 03:13:49.965904 4640 catalog_manager.cc:4841] TS a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741): tablet 123340c510f0495481c7413bfec2760d (table test-table [id=cf9107f902834222b1643e9fc40272da]) successfully deleted
I20260929 03:13:49.969936 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 4984
I20260929 03:13:49.972337 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:13:49.970199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:45887 (local address 127.0.0.1:55018)
0929 03:13:49.972137 (+ 1938us) negotiation.cc:107] Waiting for socket to connect
0929 03:13:49.972222 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.66:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1824}
I20260929 03:13:49.976406 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 4622
I20260929 03:14:09.984027 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:37661
--webserver_interface=127.20.174.126
--webserver_port=45619
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:37661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:10.103084 5151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:10.103406 5151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:10.103449 5151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:10.107602 5151 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:14:10.107702 5151 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:14:10.107719 5151 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:14:10.107733 5151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:10.107753 5151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:10.107771 5151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:10.107787 5151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:10.107803 5151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:10.113456 5151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:37661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:37661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=45619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:10.114620 5151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:10.115648 5151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:10.126142 5157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:10.126138 5156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:10.126219 5151 server_base.cc:1034] running on GCE node
W20260929 03:14:10.126135 5159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:10.127089 5151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:10.128458 5151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:10.129652 5151 hybrid_clock.cc:648] HybridClock initialized: now 1790651650129623 us; error 46 us; skew 500 ppm
Sep 29 03:14:10 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651650, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:10.133040 5151 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:14:10.134119 5151 webserver.cc:460] Webserver started at http://127.20.174.126:45619/ using document root <none> and password file <none>
I20260929 03:14:10.134783 5151 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:10.134856 5151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:10.139495 5151 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:14:10.142176 5166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:10.143977 5151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:14:10.144119 5151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21"
format_stamp: "Formatted at 2026-09-29 03:13:27 on dist-test-slave-k1dg"
server_key: "bdb9e94d2387baced4903a0be9c349f5"
server_key_iv: "6b4f1a11691ebf8a8b001302e2650a4c"
server_key_version: "encryptionkey@0"
I20260929 03:14:10.144729 5151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:10.170570 5151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:10.174906 5151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:10.175113 5151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:10.182991 5151 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:37661
I20260929 03:14:10.183408 5218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:37661 every 8 connection(s)
I20260929 03:14:10.184309 5151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:10.189179 5219 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:14:10.191120 5219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Bootstrap starting.
I20260929 03:14:10.192560 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 5151
I20260929 03:14:10.193048 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:35253
--local_ip_for_outbound_sockets=127.20.174.65
--tserver_master_addrs=127.20.174.126:37661
--webserver_port=34011
--webserver_interface=127.20.174.65
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:14:10.209811 5219 log.cc:824] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:10.222673 5219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:14:10.223140 5219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Bootstrap complete.
I20260929 03:14:10.227771 5219 raft_consensus.cc:348] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:14:10.228238 5219 raft_consensus.cc:729] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d389bb2c24cb4471b8ffbeaa0bc93b21, State: Initialized, Role: FOLLOWER
I20260929 03:14:10.228790 5219 consensus_queue.cc:260] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:14:10.228951 5219 raft_consensus.cc:388] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:10.229044 5219 raft_consensus.cc:482] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:10.229175 5219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:14:10.231527 5219 raft_consensus.cc:504] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:14:10.231935 5219 leader_election.cc:302] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d389bb2c24cb4471b8ffbeaa0bc93b21; no voters:
I20260929 03:14:10.232327 5219 leader_election.cc:288] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:14:10.232429 5223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:14:10.232633 5223 raft_consensus.cc:686] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [term 3 LEADER]: Becoming Leader. State: Replica: d389bb2c24cb4471b8ffbeaa0bc93b21, State: Running, Role: LEADER
I20260929 03:14:10.232947 5223 consensus_queue.cc:237] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } }
I20260929 03:14:10.233671 5219 sys_catalog.cc:564] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:10.234007 5223 sys_catalog.cc:455] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d389bb2c24cb4471b8ffbeaa0bc93b21. Latest consensus state: current_term: 3 leader_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } } }
I20260929 03:14:10.234133 5223 sys_catalog.cc:458] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:10.234545 5227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:10.234552 5224 sys_catalog.cc:455] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d389bb2c24cb4471b8ffbeaa0bc93b21" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37661 } } }
I20260929 03:14:10.234838 5224 sys_catalog.cc:458] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:10.241376 5227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:10.242394 5227 catalog_manager.cc:1259] Loaded cluster ID: 46e16f05f81a41329d44ce3316cf4ee0
I20260929 03:14:10.242499 5227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:10.245191 5227 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:10.246485 5227 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Loaded TSK: 0
I20260929 03:14:10.246588 5227 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d389bb2c24cb4471b8ffbeaa0bc93b21: Loaded TSK: 1
F20260929 03:14:10.247462 5227 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8c94aa79ad google::LogMessage::Fail() at ??:0
@ 0x7f8c94aaa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8c94aa74de google::LogMessage::Flush() at ??:0
@ 0x7f8c94aa8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8c9776b18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8c97763f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8c9f40c7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8c9f3e699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8c9f3e7178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f8c9f428df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8c9f44782b std::function<>::operator()() at ??:0
@ 0x7f8c9f3e6b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8c9f3e7f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8c9f3e4579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8c9f428640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8c96cf93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8c96cf9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f8c96cfb3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8c96cea512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8c9832e6db start_thread at ??:0
@ 0x7f8c935a561f clone at ??:0
W20260929 03:14:10.338483 5221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:10.338783 5221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:10.338831 5221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:10.343676 5221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:10.343729 5221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:10.343793 5221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:14:10.350374 5221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:35253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=34011
--tserver_master_addrs=127.20.174.126:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:10.351640 5221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:10.352818 5221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:10.364037 5265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:10.364048 5264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:10.364125 5268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:10.365514 5221 server_base.cc:1034] running on GCE node
I20260929 03:14:10.365949 5221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:10.366814 5221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:10.368005 5221 hybrid_clock.cc:648] HybridClock initialized: now 1790651650367928 us; error 92 us; skew 500 ppm
Sep 29 03:14:10 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651650, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:10.371546 5221 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:14:10.372670 5221 webserver.cc:460] Webserver started at http://127.20.174.65:34011/ using document root <none> and password file <none>
I20260929 03:14:10.373314 5221 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:10.373389 5221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:10.377740 5221 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:14:10.380303 5276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:10.381768 5221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:14:10.381879 5221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "fd8b6b547d594a72b4f4bcf02e504f4d"
format_stamp: "Formatted at 2026-09-29 03:13:27 on dist-test-slave-k1dg"
server_key: "079d5834a2dc57ac5d60d33145977fd8"
server_key_iv: "091e7f76fa6625b788b491cf7bdcccb7"
server_key_version: "encryptionkey@0"
I20260929 03:14:10.382634 5221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:10.397745 5221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:10.401933 5221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:10.402182 5221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:10.403139 5221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:10.404470 5283 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:14:10.409624 5221 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:14:10.409723 5221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 03:14:10.409782 5221 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:14:10.411806 5221 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:14:10.411881 5221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 03:14:10.412154 5283 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: Bootstrap starting.
I20260929 03:14:10.422880 5221 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:35253
I20260929 03:14:10.423092 5390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:35253 every 8 connection(s)
I20260929 03:14:10.424389 5221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:14:10.431577 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 5221
I20260929 03:14:10.432191 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:45887
--local_ip_for_outbound_sockets=127.20.174.66
--tserver_master_addrs=127.20.174.126:37661
--webserver_port=42225
--webserver_interface=127.20.174.66
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:14:10.434830 5283 log.cc:824] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:10.446393 5283 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:14:10.446907 5283 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: Bootstrap complete.
I20260929 03:14:10.448474 5283 ts_tablet_manager.cc:1397] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.008s
I20260929 03:14:10.452193 5283 raft_consensus.cc:348] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:14:10.452940 5283 raft_consensus.cc:729] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8b6b547d594a72b4f4bcf02e504f4d, State: Initialized, Role: FOLLOWER
I20260929 03:14:10.453531 5283 consensus_queue.cc:260] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:14:10.454726 5283 ts_tablet_manager.cc:1428] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
W20260929 03:14:10.506563 5393 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:14:10.427554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:52293)
0929 03:14:10.428313 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:10.428379 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:10.429721 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:10.505650 (+ 75929us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":194,"threads_started":1}
W20260929 03:14:10.507292 5391 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:37661 (0 consecutive failures): Network error: Failed to ping master at 127.20.174.126:37661: Client connection negotiation failed: client connection to 127.20.174.126:37661: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:14:10.508329 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:10.507970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:44645)
0929 03:14:10.508108 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:10.508259 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 03:14:10.509321 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:10.509060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:45365)
0929 03:14:10.509190 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:10.509253 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 03:14:10.509735 5391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:14:10.561460 5394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:10.561755 5394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:10.561803 5394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:10.566256 5394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:10.566341 5394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:10.566406 5394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:14:10.572243 5394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:45887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=42225
--tserver_master_addrs=127.20.174.126:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:10.573436 5394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:10.574610 5394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:10.585376 5402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:10.585421 5404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:10.585421 5401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:10.585608 5394 server_base.cc:1034] running on GCE node
I20260929 03:14:10.586203 5394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:10.586997 5394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:10.588174 5394 hybrid_clock.cc:648] HybridClock initialized: now 1790651650588145 us; error 42 us; skew 500 ppm
Sep 29 03:14:10 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651650, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:10.591260 5394 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:14:10.592236 5394 webserver.cc:460] Webserver started at http://127.20.174.66:42225/ using document root <none> and password file <none>
I20260929 03:14:10.592787 5394 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:10.592846 5394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:10.597085 5394 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:14:10.599516 5411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:10.601070 5394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:10.601263 5394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "2861338795d74860b8f4ebda76b6a400"
format_stamp: "Formatted at 2026-09-29 03:13:28 on dist-test-slave-k1dg"
server_key: "95f5ca700006673fd7d96e9a8e377414"
server_key_iv: "7b4cde3c8f90618d3f8bf02ab50fa678"
server_key_version: "encryptionkey@0"
I20260929 03:14:10.601967 5394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:10.616374 5394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:10.619803 5394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:10.619968 5394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:10.620644 5394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:10.621798 5418 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:14:10.626235 5394 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:14:10.626318 5394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 03:14:10.626379 5394 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:14:10.628489 5394 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:14:10.628543 5394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 03:14:10.628873 5418 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Bootstrap starting.
I20260929 03:14:10.639921 5394 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:45887
I20260929 03:14:10.639990 5525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:45887 every 8 connection(s)
I20260929 03:14:10.641132 5394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:14:10.645522 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:10.643447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:50195)
0929 03:14:10.644040 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:10.644227 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":131,"threads_started":1}
W20260929 03:14:10.646133 5526 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:37661 (0 consecutive failures): Network error: Failed to ping master at 127.20.174.126:37661: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
I20260929 03:14:10.647002 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:10.646700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:52529)
0929 03:14:10.646854 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:10.646923 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 03:14:10.647847 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:10.647545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:48109)
0929 03:14:10.647709 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:10.647776 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 03:14:10.648084 5526 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:14:10.649893 5418 log.cc:824] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:10.650975 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 5394
I20260929 03:14:10.651526 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:45741
--local_ip_for_outbound_sockets=127.20.174.67
--tserver_master_addrs=127.20.174.126:37661
--webserver_port=35377
--webserver_interface=127.20.174.67
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:14:10.658380 5418 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:14:10.658842 5418 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Bootstrap complete.
I20260929 03:14:10.660015 5418 ts_tablet_manager.cc:1397] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Time spent bootstrapping tablet: real 0.031s user 0.016s sys 0.013s
I20260929 03:14:10.662926 5418 raft_consensus.cc:348] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:14:10.663376 5418 raft_consensus.cc:729] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2861338795d74860b8f4ebda76b6a400, State: Initialized, Role: FOLLOWER
I20260929 03:14:10.663889 5418 consensus_queue.cc:260] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:14:10.665045 5418 ts_tablet_manager.cc:1428] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 03:14:10.665305 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:10.664979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:51747)
0929 03:14:10.665133 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:10.665206 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 03:14:10.666064 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:10.665835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:59115)
0929 03:14:10.665949 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:10.666001 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 03:14:10.772753 5530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:10.773069 5530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:10.773118 5530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:10.777503 5530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:10.777596 5530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:10.777661 5530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:14:10.784124 5530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:45741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=35377
--tserver_master_addrs=127.20.174.126:37661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:10.785356 5530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:10.786602 5530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:10.797550 5537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:10.797531 5536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:10.797550 5539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:10.798192 5530 server_base.cc:1034] running on GCE node
I20260929 03:14:10.798733 5530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:10.799465 5530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:10.800669 5530 hybrid_clock.cc:648] HybridClock initialized: now 1790651650800621 us; error 69 us; skew 500 ppm
Sep 29 03:14:10 dist-test-slave-k1dg krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651650, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:10.803932 5530 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:14:10.805027 5530 webserver.cc:460] Webserver started at http://127.20.174.67:35377/ using document root <none> and password file <none>
I20260929 03:14:10.805694 5530 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:10.805804 5530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:10.810587 5530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:14:10.813159 5546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:10.814913 5530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:10.815083 5530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "a84bac3401714a35a403c5c6d177dacb"
format_stamp: "Formatted at 2026-09-29 03:13:28 on dist-test-slave-k1dg"
server_key: "a680bd30d5dd4422e86c7486c0130ece"
server_key_iv: "a5579bc8bd436b17370d3cfbd329bde4"
server_key_version: "encryptionkey@0"
I20260929 03:14:10.815785 5530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:10.837100 5530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:10.840832 5530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:10.841094 5530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:10.841787 5530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:10.842852 5530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:10.842938 5530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:10.843021 5530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:10.843075 5530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:10.853410 5530 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:45741
I20260929 03:14:10.853430 5659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:45741 every 8 connection(s)
I20260929 03:14:10.854650 5530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:14:10.859427 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:10.857271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:46349)
0929 03:14:10.857858 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:10.857992 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":125,"threads_started":1}
I20260929 03:14:10.859671 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 5530
W20260929 03:14:10.860029 5660 heartbeater.cc:644] Failed to heartbeat to 127.20.174.126:37661 (0 consecutive failures): Network error: Failed to ping master at 127.20.174.126:37661: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
I20260929 03:14:10.860744 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:10.860437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:37861)
0929 03:14:10.860568 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:10.860667 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 03:14:10.861510 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:10.861309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:33141)
0929 03:14:10.861412 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:10.861455 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:14:10.861822 5660 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:14:11.511188 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:11.510633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:53405)
0929 03:14:11.511012 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:11.511079 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":161,"threads_started":1}
I20260929 03:14:11.667476 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:11.666990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:49045)
0929 03:14:11.667257 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:11.667336 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260929 03:14:11.863108 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:11.862598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:47727)
0929 03:14:11.862917 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:11.862980 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":83,"threads_started":1}
I20260929 03:14:11.866534 5667 raft_consensus.cc:482] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:14:11.866739 5667 raft_consensus.cc:504] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:14:11.867993 5667 leader_election.cc:288] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741), fd8b6b547d594a72b4f4bcf02e504f4d (127.20.174.65:35253)
Sep 29 03:14:11 dist-test-slave-k1dg krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651650, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.67@KRBTEST.COM
Sep 29 03:14:11 dist-test-slave-k1dg krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651650, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.65@KRBTEST.COM
I20260929 03:14:11.890066 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:11.868825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:59107 (local address 127.20.174.65:35253)
0929 03:14:11.869461 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:11.869466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:11.869489 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:14:11.869613 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:11.869617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:11.869894 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:11.870047 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:11.871665 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:11.872668 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:11.873953 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:11.874502 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:11.874691 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:11.884774 (+ 10083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:11.884782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:11.884796 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:11.884832 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:11.887049 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:11.887782 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:11.887796 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:11.887802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:11.887998 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:11.888425 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:11.888428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:11.888429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:11.888855 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:11.889014 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:11.889592 (+ 578us) server_negotiation.cc:299] Negotiation successful
0929 03:14:11.889870 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":91,"threads_started":1}
I20260929 03:14:11.890064 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:11.868326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:45741 (local address 127.20.174.66:35099)
0929 03:14:11.868976 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:11.868991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:11.869097 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:11.870826 (+ 1729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:11.870829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:11.870987 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:11.871454 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:11.871472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:11.872838 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:11.872844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:11.873791 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:11.873802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:11.874639 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:11.874643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:11.874903 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:11.881896 (+ 6993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:11.881919 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:11.884155 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:11.887072 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:11.887080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:11.887097 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:11.887605 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:11.888070 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:11.888076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:11.888079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:11.888248 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:11.888993 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:11.889000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:11.889253 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 03:14:11.889473 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 03:14:11.889880 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":236,"spinlock_wait_cycles":4864,"thread_start_us":91,"threads_started":1}
I20260929 03:14:11.890127 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:11.868571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:35099 (local address 127.20.174.67:45741)
0929 03:14:11.869260 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:11.869284 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:11.869325 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:14:11.870058 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:11.870067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:11.870504 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:11.870857 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:11.871689 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:11.872671 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:11.873951 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:11.874498 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:11.874692 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:11.884554 (+ 9862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:11.884564 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:11.884589 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:11.884639 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:11.886921 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:11.887804 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:11.887810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:11.887815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:11.887953 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:11.888376 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:11.888381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:11.888383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:11.888857 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:11.889012 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:11.889592 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 03:14:11.889909 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":107,"threads_started":1}
I20260929 03:14:11.890955 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:11.868326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:35253 (local address 127.20.174.66:59107)
0929 03:14:11.868493 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:11.868527 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:11.869022 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:11.870232 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:11.870242 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:11.870825 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:11.871454 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:11.871472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:11.872933 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:11.872936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:11.873795 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:11.873805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:11.874637 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:11.874640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:11.874901 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:11.881894 (+ 6993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:11.881925 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:11.884531 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:11.887188 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:11.887190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:11.887192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:11.887661 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:11.888125 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:11.888127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:11.888128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:11.888269 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:11.888991 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:11.888998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:11.889253 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 03:14:11.889473 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 03:14:11.889971 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":347,"spinlock_wait_cycles":46464}
I20260929 03:14:11.892277 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123340c510f0495481c7413bfec2760d" candidate_uuid: "2861338795d74860b8f4ebda76b6a400" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" is_pre_election: true
I20260929 03:14:11.892639 5345 raft_consensus.cc:2454] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2861338795d74860b8f4ebda76b6a400 in term 1.
W20260929 03:14:11.893318 5413 leader_election.cc:341] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741): Not found: Tablet not found: 123340c510f0495481c7413bfec2760d
I20260929 03:14:11.893543 5414 leader_election.cc:302] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2861338795d74860b8f4ebda76b6a400, fd8b6b547d594a72b4f4bcf02e504f4d; no voters: a84bac3401714a35a403c5c6d177dacb
I20260929 03:14:11.893895 5667 raft_consensus.cc:2781] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 03:14:11.893985 5667 raft_consensus.cc:482] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:14:11.894016 5667 raft_consensus.cc:3037] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:14:11.896436 5667 raft_consensus.cc:504] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:14:11.896862 5667 leader_election.cc:288] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [CANDIDATE]: Term 2 election: Requested vote from peers a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741), fd8b6b547d594a72b4f4bcf02e504f4d (127.20.174.65:35253)
I20260929 03:14:11.897454 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123340c510f0495481c7413bfec2760d" candidate_uuid: "2861338795d74860b8f4ebda76b6a400" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d"
I20260929 03:14:11.897622 5345 raft_consensus.cc:3037] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 1 FOLLOWER]: Advancing to term 2
W20260929 03:14:11.897758 5413 leader_election.cc:341] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741): Not found: Tablet not found: 123340c510f0495481c7413bfec2760d
I20260929 03:14:11.900092 5345 raft_consensus.cc:2454] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2861338795d74860b8f4ebda76b6a400 in term 2.
I20260929 03:14:11.900535 5414 leader_election.cc:302] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2861338795d74860b8f4ebda76b6a400, fd8b6b547d594a72b4f4bcf02e504f4d; no voters: a84bac3401714a35a403c5c6d177dacb
I20260929 03:14:11.900763 5667 raft_consensus.cc:2781] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:14:11.900974 5667 raft_consensus.cc:686] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [term 2 LEADER]: Becoming Leader. State: Replica: 2861338795d74860b8f4ebda76b6a400, State: Running, Role: LEADER
I20260929 03:14:11.901463 5667 consensus_queue.cc:237] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:14:11.901978 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:11.901692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:56993)
0929 03:14:11.901838 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:11.901903 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 03:14:12.380988 5345 raft_consensus.cc:1261] T 123340c510f0495481c7413bfec2760d P fd8b6b547d594a72b4f4bcf02e504f4d [term 2 FOLLOWER]: Refusing update from remote peer 2861338795d74860b8f4ebda76b6a400: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 03:14:12.381928 5667 consensus_queue.cc:1035] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 03:14:12.384536 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:12.384067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:44363)
0929 03:14:12.384314 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:12.384400 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
W20260929 03:14:12.386308 5413 consensus_peers.cc:487] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400 -> Peer a84bac3401714a35a403c5c6d177dacb (127.20.174.67:45741): Couldn't send request to peer a84bac3401714a35a403c5c6d177dacb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 123340c510f0495481c7413bfec2760d. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:14:12.387451 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:12.387127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:59805)
0929 03:14:12.387283 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:12.387356 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 03:14:12.387966 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:12.387664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:58725)
0929 03:14:12.387827 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:12.387886 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 03:14:12.864737 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:12.864143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:36567)
0929 03:14:12.864527 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:12.864598 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":149,"threads_started":1}
I20260929 03:14:12.887533 5679 ts_tablet_manager.cc:927] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Initiating tablet copy from peer 2861338795d74860b8f4ebda76b6a400 (127.20.174.66:45887)
I20260929 03:14:12.888376 5679 tablet_copy_client.cc:316] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: tablet copy: Beginning tablet copy session from remote peer at address 127.20.174.66:45887
Sep 29 03:14:12 dist-test-slave-k1dg krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651650, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.66@KRBTEST.COM
I20260929 03:14:12.900262 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:12.889093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:52185 (local address 127.20.174.66:45887)
0929 03:14:12.889342 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:12.889351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:12.889392 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:14:12.889444 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:12.889448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:12.889638 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:12.889761 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:12.890486 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:12.891225 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:12.892185 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:12.892581 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:12.892678 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:12.895967 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:12.895979 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:12.895993 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:12.896024 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:12.897892 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:12.898508 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:12.898515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:12.898516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:12.898618 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:12.898988 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:12.898992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:12.898998 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:12.899351 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:12.899519 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:12.899926 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 03:14:12.900100 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260929 03:14:12.900254 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:12.888930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:45887 (local address 127.20.174.67:52185)
0929 03:14:12.889082 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:12.889104 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:12.889251 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:12.889794 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:12.889800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:12.890194 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:12.890312 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:12.890319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:12.891352 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:12.891355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:12.892047 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:12.892058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:12.892840 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:12.892844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:12.892966 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:12.893709 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:12.893726 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:12.895664 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:12.898042 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:12.898054 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:12.898062 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:12.898341 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:12.898744 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:12.898747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:12.898750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:12.898849 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:12.899488 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:12.899506 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:12.899655 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 03:14:12.899832 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 03:14:12.900059 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 03:14:12.901552 5500 tablet_copy_service.cc:140] P 2861338795d74860b8f4ebda76b6a400: Received BeginTabletCopySession request for tablet 123340c510f0495481c7413bfec2760d from peer a84bac3401714a35a403c5c6d177dacb ({username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:52185)
I20260929 03:14:12.901716 5500 tablet_copy_service.cc:161] P 2861338795d74860b8f4ebda76b6a400: Beginning new tablet copy session on tablet 123340c510f0495481c7413bfec2760d from peer a84bac3401714a35a403c5c6d177dacb at {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:52185: session id = a84bac3401714a35a403c5c6d177dacb-123340c510f0495481c7413bfec2760d
I20260929 03:14:12.902951 5500 tablet_copy_source_session.cc:205] T 123340c510f0495481c7413bfec2760d P 2861338795d74860b8f4ebda76b6a400: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:14:12.907596 5679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123340c510f0495481c7413bfec2760d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:12.915328 5679 tablet_copy_client.cc:791] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: tablet copy: Starting download of 0 data blocks...
I20260929 03:14:12.915580 5679 tablet_copy_client.cc:655] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: tablet copy: Starting download of 1 WAL segments...
I20260929 03:14:12.917728 5679 tablet_copy_client.cc:531] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:14:12.919777 5679 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Bootstrap starting.
I20260929 03:14:12.925774 5679 log.cc:824] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:12.933974 5679 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:14:12.934369 5679 tablet_bootstrap.cc:492] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Bootstrap complete.
I20260929 03:14:12.934808 5679 ts_tablet_manager.cc:1397] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Time spent bootstrapping tablet: real 0.015s user 0.015s sys 0.000s
I20260929 03:14:12.937389 5679 raft_consensus.cc:348] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:14:12.937722 5679 raft_consensus.cc:729] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84bac3401714a35a403c5c6d177dacb, State: Initialized, Role: FOLLOWER
I20260929 03:14:12.938237 5679 consensus_queue.cc:260] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2861338795d74860b8f4ebda76b6a400" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 45887 } } peers { permanent_uuid: "a84bac3401714a35a403c5c6d177dacb" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 45741 } } peers { permanent_uuid: "fd8b6b547d594a72b4f4bcf02e504f4d" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 35253 } }
I20260929 03:14:12.939407 5679 ts_tablet_manager.cc:1428] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:14:12.939754 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:12.939455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:45115)
0929 03:14:12.939583 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:12.939642 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 03:14:12.940238 5500 tablet_copy_service.cc:342] P 2861338795d74860b8f4ebda76b6a400: Request end of tablet copy session a84bac3401714a35a403c5c6d177dacb-123340c510f0495481c7413bfec2760d received from {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:52185
I20260929 03:14:12.940371 5500 tablet_copy_service.cc:433] P 2861338795d74860b8f4ebda76b6a400: ending tablet copy session a84bac3401714a35a403c5c6d177dacb-123340c510f0495481c7413bfec2760d on tablet 123340c510f0495481c7413bfec2760d with peer a84bac3401714a35a403c5c6d177dacb
I20260929 03:14:12.941031 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:12.940303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:45897)
0929 03:14:12.940643 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:12.940936 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:14:13.376479 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:13.375982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:42907)
0929 03:14:13.376247 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:13.376325 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260929 03:14:13.376410 5614 raft_consensus.cc:1203] T 123340c510f0495481c7413bfec2760d P a84bac3401714a35a403c5c6d177dacb [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260929 03:14:13.377971 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:13.377420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:56479)
0929 03:14:13.377780 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:13.377866 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 03:14:13.389521 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:13.388854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:49737)
0929 03:14:13.389307 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:13.389390 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":92,"threads_started":1}
I20260929 03:14:14.377944 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:14.377377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:54731)
0929 03:14:14.377692 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:14.377837 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260929 03:14:14.379230 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:14.378756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:40951)
0929 03:14:14.379089 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:14.379144 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":98,"threads_started":1}
I20260929 03:14:14.391005 5689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:14.390529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:54607)
0929 03:14:14.390807 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:14.390880 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260929 03:14:15.379441 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:15.378905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:36107)
0929 03:14:15.379242 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:15.379308 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260929 03:14:15.380632 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:15.380191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:57075)
0929 03:14:15.380481 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:15.380546 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":97,"threads_started":1}
I20260929 03:14:15.392321 5692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:15.391903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:54891)
0929 03:14:15.392165 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:15.392222 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260929 03:14:16.380882 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:16.380427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:43963)
0929 03:14:16.380707 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:16.380770 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260929 03:14:16.381788 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:16.381510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:48961)
0929 03:14:16.381638 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:16.381696 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:14:16.382139 5695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:16.381681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:53319)
0929 03:14:16.381995 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:16.382059 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":78,"threads_started":1}
I20260929 03:14:16.393784 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:16.393326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:36333)
0929 03:14:16.393625 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:16.393687 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":73,"threads_started":1}
I20260929 03:14:17.383463 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:17.382996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:42093)
0929 03:14:17.383293 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:17.383354 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260929 03:14:17.383888 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:17.382919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:45909)
0929 03:14:17.383697 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:17.383768 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":106,"threads_started":1}
I20260929 03:14:17.395097 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:17.394665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:47651)
0929 03:14:17.394933 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:17.395001 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20260929 03:14:18.385090 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:18.384586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:44053)
0929 03:14:18.384895 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:18.384973 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":100,"threads_started":1}
I20260929 03:14:18.385090 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:18.384586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:60313)
0929 03:14:18.384895 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:18.384973 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
I20260929 03:14:18.396498 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:18.396018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:54545)
0929 03:14:18.396335 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:18.396401 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":80,"threads_started":1}
I20260929 03:14:19.386576 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:19.386027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:43573)
0929 03:14:19.386349 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:19.386419 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":155,"threads_started":1}
I20260929 03:14:19.386624 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:19.386014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:43843)
0929 03:14:19.386306 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:19.386492 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":108,"threads_started":1}
I20260929 03:14:19.397810 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:19.397355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:40485)
0929 03:14:19.397645 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:19.397709 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":96,"threads_started":1}
I20260929 03:14:20.388233 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:20.387580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:56589)
0929 03:14:20.388044 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:20.388108 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":118,"threads_started":1}
I20260929 03:14:20.388317 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:20.387580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:43037)
0929 03:14:20.387924 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:20.388157 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
I20260929 03:14:20.399230 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:20.398728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:58425)
0929 03:14:20.399068 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:20.399131 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":99,"threads_started":1}
I20260929 03:14:21.389590 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:21.389113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:49073)
0929 03:14:21.389375 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:21.389463 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
I20260929 03:14:21.389721 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:21.389240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:52719)
0929 03:14:21.389571 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:21.389618 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":86,"threads_started":1}
I20260929 03:14:21.400902 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:21.400385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:44417)
0929 03:14:21.400691 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:21.400754 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260929 03:14:22.391088 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:22.390662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:33469)
0929 03:14:22.390915 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:22.390993 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
I20260929 03:14:22.391090 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:22.390614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:57337)
0929 03:14:22.390924 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:22.390986 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":113,"threads_started":1}
I20260929 03:14:22.402400 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:22.401945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:36567)
0929 03:14:22.402235 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:22.402298 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
I20260929 03:14:23.392408 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:23.391989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:49181)
0929 03:14:23.392257 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:23.392320 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
I20260929 03:14:23.392407 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:23.391955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:37091)
0929 03:14:23.392232 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:23.392305 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
I20260929 03:14:23.403901 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:23.403305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:43953)
0929 03:14:23.403646 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:23.403800 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
I20260929 03:14:24.393814 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:24.393284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:39433)
0929 03:14:24.393614 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:24.393694 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":84,"threads_started":1}
I20260929 03:14:24.393827 5719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:24.393371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:34287)
0929 03:14:24.393682 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:24.393739 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":85,"threads_started":1}
I20260929 03:14:24.405313 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:24.404707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:45775)
0929 03:14:24.405057 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:24.405114 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":179,"threads_started":1}
I20260929 03:14:25.395346 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:25.394744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.66:60101)
0929 03:14:25.395112 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:25.395204 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
I20260929 03:14:25.395339 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:25.394754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.67:50071)
0929 03:14:25.395194 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:25.395249 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":137,"threads_started":1}
I20260929 03:14:25.406809 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:25.406290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:37661 (local address 127.20.174.65:38397)
0929 03:14:25.406600 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:25.406673 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.126:37661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":152,"threads_started":1}
I20260929 03:14:25.862203 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 5221
I20260929 03:14:25.868297 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 5394
I20260929 03:14:25.874051 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 5530
2026-09-29T03:14:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59746 ms)
[----------] 30 tests from SecurityITest (252363 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:25 dist-test-slave-k1dg krb5kdc[5729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:25 dist-test-slave-k1dg krb5kdc[5729](info): set up 2 sockets
Sep 29 03:14:25 dist-test-slave-k1dg krb5kdc[5729](info): commencing operation
krb5kdc: starting...
W20260929 03:14:26.958546 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:27 dist-test-slave-k1dg krb5kdc[5729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:14:27.100093 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:42909
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:42909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:27.222489 5742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:27.222838 5742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:27.222890 5742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:27.227845 5742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:27.227936 5742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:27.227955 5742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:27.227974 5742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:27.227990 5742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:27.234534 5742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:42909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:42909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:27.235786 5742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:27.236974 5742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:27.247434 5748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:27.247539 5747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:27.247454 5750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:27.248013 5742 server_base.cc:1034] running on GCE node
I20260929 03:14:27.248740 5742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:27.250109 5742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:27.251345 5742 hybrid_clock.cc:648] HybridClock initialized: now 1790651667251330 us; error 82 us; skew 500 ppm
Sep 29 03:14:27 dist-test-slave-k1dg krb5kdc[5729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651667, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:27.254556 5742 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:14:27.255676 5742 webserver.cc:460] Webserver started at http://127.20.174.126:33585/ using document root <none> and password file <none>
I20260929 03:14:27.256392 5742 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:27.256502 5742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:27.256768 5742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:27.259312 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "8f3794eb755b4958ae6fcfab68475a81"
format_stamp: "Formatted at 2026-09-29 03:14:27 on dist-test-slave-k1dg"
server_key: "07ff562ac411799c96dc458b5bb5e110"
server_key_iv: "c966f482ddadc7acd5a28c5e019325bf"
server_key_version: "encryptionkey@0"
I20260929 03:14:27.259991 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "8f3794eb755b4958ae6fcfab68475a81"
format_stamp: "Formatted at 2026-09-29 03:14:27 on dist-test-slave-k1dg"
server_key: "07ff562ac411799c96dc458b5bb5e110"
server_key_iv: "c966f482ddadc7acd5a28c5e019325bf"
server_key_version: "encryptionkey@0"
I20260929 03:14:27.264334 5742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:14:27.267354 5757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:27.269066 5742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:27.269259 5742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "8f3794eb755b4958ae6fcfab68475a81"
format_stamp: "Formatted at 2026-09-29 03:14:27 on dist-test-slave-k1dg"
server_key: "07ff562ac411799c96dc458b5bb5e110"
server_key_iv: "c966f482ddadc7acd5a28c5e019325bf"
server_key_version: "encryptionkey@0"
I20260929 03:14:27.269450 5742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:27.286265 5742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:27.290257 5742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:27.290575 5742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:27.299026 5742 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:42909
I20260929 03:14:27.299038 5809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:42909 every 8 connection(s)
I20260929 03:14:27.300191 5742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:27.303411 5810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:27.308444 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 5742
I20260929 03:14:27.308566 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:27.308992 21177 external_mini_cluster.cc:1428] Setting key 2dd57c00ee3b53b6bcf66fa1719fcb3a
I20260929 03:14:27.309507 5810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81: Bootstrap starting.
I20260929 03:14:27.312419 5810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:27.313319 5810 log.cc:824] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:27.315678 5810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81: No bootstrap required, opened a new log
Sep 29 03:14:27 dist-test-slave-k1dg krb5kdc[5729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:14:27.319780 5810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3794eb755b4958ae6fcfab68475a81" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42909 } }
I20260929 03:14:27.320127 5810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:27.320218 5810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3794eb755b4958ae6fcfab68475a81, State: Initialized, Role: FOLLOWER
I20260929 03:14:27.320799 5810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3794eb755b4958ae6fcfab68475a81" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42909 } }
I20260929 03:14:27.321009 5810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:27.321077 5810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:27.321189 5810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:27.322455 5810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3794eb755b4958ae6fcfab68475a81" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42909 } }
I20260929 03:14:27.322856 5810 leader_election.cc:302] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f3794eb755b4958ae6fcfab68475a81; no voters:
I20260929 03:14:27.323210 5810 leader_election.cc:288] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:27.323392 5815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:27.323664 5815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [term 1 LEADER]: Becoming Leader. State: Replica: 8f3794eb755b4958ae6fcfab68475a81, State: Running, Role: LEADER
I20260929 03:14:27.324118 5815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3794eb755b4958ae6fcfab68475a81" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42909 } }
I20260929 03:14:27.324608 5810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:27.325793 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:27.310607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.20.174.126:42909)
0929 03:14:27.311057 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:27.311077 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:27.311122 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:14:27.312034 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:27.312058 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:27.312525 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:27.312929 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:27.313682 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:27.315115 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:27.316135 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:27.316836 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:27.317003 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:27.319666 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:27.319673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:27.319690 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:27.319725 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:27.321902 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:27.322379 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:27.322386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:27.322393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:27.322568 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:27.322862 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:27.322866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:27.322871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:27.323457 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:27.324263 (+ 806us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:27.324420 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 03:14:27.324724 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":112,"threads_started":1}
I20260929 03:14:27.325785 5816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f3794eb755b4958ae6fcfab68475a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3794eb755b4958ae6fcfab68475a81" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42909 } } }
I20260929 03:14:27.326195 5816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:27.325968 5815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f3794eb755b4958ae6fcfab68475a81. Latest consensus state: current_term: 1 leader_uuid: "8f3794eb755b4958ae6fcfab68475a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3794eb755b4958ae6fcfab68475a81" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42909 } } }
I20260929 03:14:27.326297 5815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:27.327214 5824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:27.330194 5824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:27.336743 5824 catalog_manager.cc:1347] Generated new cluster ID: 5eab5c1a77ea4cd091b548cfe718f4fd
I20260929 03:14:27.336915 5824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:27.363444 5824 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:27.364530 5824 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:27.369692 5824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81: Generated new TSK 0
I20260929 03:14:27.370538 5824 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:14:27.410689 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:42909
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:27.529287 5836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:27.529577 5836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:27.529623 5836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:27.533855 5836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:27.533933 5836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:27.533995 5836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:14:27.539773 5836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:42909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:27.540898 5836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:27.541934 5836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:27.551774 5842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:27.551899 5844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:27.551899 5841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:27.552199 5836 server_base.cc:1034] running on GCE node
I20260929 03:14:27.552628 5836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:27.553288 5836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:27.554622 5836 hybrid_clock.cc:648] HybridClock initialized: now 1790651667554603 us; error 32 us; skew 500 ppm
Sep 29 03:14:27 dist-test-slave-k1dg krb5kdc[5729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651667, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:27.557535 5836 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:14:27.558575 5836 webserver.cc:460] Webserver started at http://127.20.174.65:41179/ using document root <none> and password file <none>
I20260929 03:14:27.559149 5836 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:27.559247 5836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:27.559484 5836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:27.561527 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "a12e5ef71af44c6ab9237ef1528e7594"
format_stamp: "Formatted at 2026-09-29 03:14:27 on dist-test-slave-k1dg"
server_key: "a2e4bcb3b747eacd04c6e6950ff7534c"
server_key_iv: "73d4704be4669f46e035b75b5e51840d"
server_key_version: "encryptionkey@0"
I20260929 03:14:27.562095 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "a12e5ef71af44c6ab9237ef1528e7594"
format_stamp: "Formatted at 2026-09-29 03:14:27 on dist-test-slave-k1dg"
server_key: "a2e4bcb3b747eacd04c6e6950ff7534c"
server_key_iv: "73d4704be4669f46e035b75b5e51840d"
server_key_version: "encryptionkey@0"
I20260929 03:14:27.566157 5836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 03:14:27.568805 5851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:27.570086 5836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:14:27.570230 5836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "a12e5ef71af44c6ab9237ef1528e7594"
format_stamp: "Formatted at 2026-09-29 03:14:27 on dist-test-slave-k1dg"
server_key: "a2e4bcb3b747eacd04c6e6950ff7534c"
server_key_iv: "73d4704be4669f46e035b75b5e51840d"
server_key_version: "encryptionkey@0"
I20260929 03:14:27.570379 5836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:27.589347 5836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:27.592510 5836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:27.592715 5836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:27.593310 5836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:27.594141 5836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:27.594209 5836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:27.594290 5836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:27.594347 5836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:27.603404 5836 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:44227
I20260929 03:14:27.603425 5964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:44227 every 8 connection(s)
I20260929 03:14:27.604316 5836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:14:27.607653 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 5836
I20260929 03:14:27.607817 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:14:27.608153 21177 external_mini_cluster.cc:1428] Setting key 88ce96999d6dc0e72eecccbf25dd7966
Sep 29 03:14:27 dist-test-slave-k1dg krb5kdc[5729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651667, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:14:27.619531 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:27.606176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:33583 (local address 127.20.174.126:42909)
0929 03:14:27.606351 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:27.606355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:27.607159 (+ 804us) server_negotiation.cc:407] Connection header received
0929 03:14:27.608732 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:27.608736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:27.608796 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:27.608906 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:27.610107 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:27.610727 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:27.612506 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:27.612945 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:27.613069 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:27.615697 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:27.615704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:27.615707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:27.615739 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:27.617387 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:27.617884 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:27.617889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:27.617894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:27.617987 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:27.618315 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:27.618321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:27.618324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:27.618603 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:27.618719 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:27.619217 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 03:14:27.619367 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:14:27.620352 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:27.606567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:42909 (local address 127.20.174.65:33583)
0929 03:14:27.607009 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:27.607052 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:27.608505 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:27.609086 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:27.609093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:27.609449 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:27.609967 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:27.609981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:27.610856 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:27.610860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:27.612370 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:27.612380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:27.613109 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:27.613112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:27.613268 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:27.613924 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:27.613941 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:27.615520 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:27.617507 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:27.617511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:27.617524 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:27.617773 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:27.618084 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:27.618086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:27.618087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:27.618218 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:27.618714 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:27.618719 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:27.618995 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 03:14:27.619162 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 03:14:27.619428 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:14:27.621819 5965 heartbeater.cc:347] Connected to a master server at 127.20.174.126:42909
I20260929 03:14:27.622097 5965 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:27.622714 5965 heartbeater.cc:511] Master 127.20.174.126:42909 requested a full tablet report, sending...
I20260929 03:14:27.624603 5772 ts_manager.cc:194] Registered new tserver with Master: a12e5ef71af44c6ab9237ef1528e7594 (127.20.174.65:44227)
I20260929 03:14:27.626009 5772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:33583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:14:27.637804 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:42909
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:27.757402 5970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:27.757704 5970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:27.757748 5970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:27.762204 5970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:27.762290 5970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:27.762393 5970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:14:27.768210 5970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:42909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:27.769277 5970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:27.770262 5970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:27.780429 5978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:27.780429 5975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:27.780526 5970 server_base.cc:1034] running on GCE node
W20260929 03:14:27.780431 5976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:27.781114 5970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:27.781821 5970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:27.783008 5970 hybrid_clock.cc:648] HybridClock initialized: now 1790651667782974 us; error 57 us; skew 500 ppm
Sep 29 03:14:27 dist-test-slave-k1dg krb5kdc[5729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651667, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:27.785820 5970 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:14:27.786803 5970 webserver.cc:460] Webserver started at http://127.20.174.66:41121/ using document root <none> and password file <none>
I20260929 03:14:27.787348 5970 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:27.787405 5970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:27.787570 5970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:27.789750 5970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637"
format_stamp: "Formatted at 2026-09-29 03:14:27 on dist-test-slave-k1dg"
server_key: "f8ab560b8e2ad4d0bc1e1ed07ac81e2c"
server_key_iv: "f5c758398c4876ac231499467959f463"
server_key_version: "encryptionkey@0"
I20260929 03:14:27.790294 5970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637"
format_stamp: "Formatted at 2026-09-29 03:14:27 on dist-test-slave-k1dg"
server_key: "f8ab560b8e2ad4d0bc1e1ed07ac81e2c"
server_key_iv: "f5c758398c4876ac231499467959f463"
server_key_version: "encryptionkey@0"
I20260929 03:14:27.794121 5970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 03:14:27.796617 5985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:27.797967 5970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:14:27.798087 5970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637"
format_stamp: "Formatted at 2026-09-29 03:14:27 on dist-test-slave-k1dg"
server_key: "f8ab560b8e2ad4d0bc1e1ed07ac81e2c"
server_key_iv: "f5c758398c4876ac231499467959f463"
server_key_version: "encryptionkey@0"
I20260929 03:14:27.798195 5970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:27.815976 5970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:27.819176 5970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:27.819346 5970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:27.819885 5970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:27.820684 5970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:27.820731 5970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:27.820770 5970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:27.820785 5970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:27.830641 5970 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:39323
I20260929 03:14:27.830660 6098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:39323 every 8 connection(s)
I20260929 03:14:27.831606 5970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:14:27.834849 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 5970
I20260929 03:14:27.834981 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:14:27.835304 21177 external_mini_cluster.cc:1428] Setting key d2817c21a400fefa963434fa50e23406
Sep 29 03:14:27 dist-test-slave-k1dg krb5kdc[5729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651667, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:14:27.847569 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:27.833569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:52531 (local address 127.20.174.126:42909)
0929 03:14:27.833716 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:27.833721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:27.835242 (+ 1521us) server_negotiation.cc:407] Connection header received
0929 03:14:27.836335 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:27.836339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:27.836397 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:27.836486 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:27.837816 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:27.838743 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:27.839938 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:27.840410 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:27.840540 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:27.843092 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:27.843099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:27.843103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:27.843141 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:27.845050 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:27.845579 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:27.845583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:27.845585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:27.845682 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:27.846174 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:27.846180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:27.846192 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:27.846423 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:27.846577 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:27.847220 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 03:14:27.847359 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:14:27.848482 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:27.833909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:42909 (local address 127.20.174.66:52531)
0929 03:14:27.835054 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:27.835104 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:27.836106 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:27.836705 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:27.836714 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:27.837110 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:27.837642 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:27.837658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:27.838881 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:27.838884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:27.839800 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:27.839809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:27.840573 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:27.840575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:27.840724 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:27.841367 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:27.841383 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:27.842902 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:27.845176 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:27.845181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:27.845196 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:27.845471 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:27.845811 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:27.845814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:27.845815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:27.846061 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:27.846587 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:27.846630 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:27.846948 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 03:14:27.847175 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 03:14:27.847539 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":653,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:14:27.849767 6099 heartbeater.cc:347] Connected to a master server at 127.20.174.126:42909
I20260929 03:14:27.850054 6099 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:27.850737 6099 heartbeater.cc:511] Master 127.20.174.126:42909 requested a full tablet report, sending...
I20260929 03:14:27.852072 5772 ts_manager.cc:194] Registered new tserver with Master: 1b44d3f1a1254ceaa1a0ab585d0a3637 (127.20.174.66:39323)
I20260929 03:14:27.853044 5772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:52531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:14:27.867542 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:42909
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:27.982666 6104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:27.982962 6104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:27.983007 6104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:27.987360 6104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:27.987452 6104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:27.987550 6104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:14:27.993295 6104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:42909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:27.994520 6104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:27.995618 6104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:28.005679 6110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:28.005699 6109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:28.005904 6104 server_base.cc:1034] running on GCE node
W20260929 03:14:28.005699 6112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:28.006482 6104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:28.007217 6104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:28.008432 6104 hybrid_clock.cc:648] HybridClock initialized: now 1790651668008412 us; error 43 us; skew 500 ppm
Sep 29 03:14:28 dist-test-slave-k1dg krb5kdc[5729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651668, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:28.011562 6104 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:14:28.012629 6104 webserver.cc:460] Webserver started at http://127.20.174.67:45027/ using document root <none> and password file <none>
I20260929 03:14:28.013240 6104 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:28.013350 6104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:28.013594 6104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:28.015817 6104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "1a039bad31984ebd84c8839ca4446128"
format_stamp: "Formatted at 2026-09-29 03:14:28 on dist-test-slave-k1dg"
server_key: "5c5c16d5fa8a7caefc707200bf5b235e"
server_key_iv: "c1ab13d9df0fa57887fa9e6874578332"
server_key_version: "encryptionkey@0"
I20260929 03:14:28.016408 6104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "1a039bad31984ebd84c8839ca4446128"
format_stamp: "Formatted at 2026-09-29 03:14:28 on dist-test-slave-k1dg"
server_key: "5c5c16d5fa8a7caefc707200bf5b235e"
server_key_iv: "c1ab13d9df0fa57887fa9e6874578332"
server_key_version: "encryptionkey@0"
I20260929 03:14:28.020520 6104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:14:28.023190 6119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:28.024513 6104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:28.024652 6104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "1a039bad31984ebd84c8839ca4446128"
format_stamp: "Formatted at 2026-09-29 03:14:28 on dist-test-slave-k1dg"
server_key: "5c5c16d5fa8a7caefc707200bf5b235e"
server_key_iv: "c1ab13d9df0fa57887fa9e6874578332"
server_key_version: "encryptionkey@0"
I20260929 03:14:28.024771 6104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:28.049243 6104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:28.052852 6104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:28.053081 6104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:28.053750 6104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:28.054720 6104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:28.054792 6104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:28.054862 6104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:28.054911 6104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:28.065130 6104 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:33173
I20260929 03:14:28.065148 6232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:33173 every 8 connection(s)
I20260929 03:14:28.066141 6104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:14:28.075230 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 6104
I20260929 03:14:28.075448 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:14:28.075841 21177 external_mini_cluster.cc:1428] Setting key 76763cffd0a05684d65a582a95710974
Sep 29 03:14:28 dist-test-slave-k1dg krb5kdc[5729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651668, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:14:28.082028 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.068201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:45261 (local address 127.20.174.126:42909)
0929 03:14:28.068463 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:28.068468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:28.069808 (+ 1340us) server_negotiation.cc:407] Connection header received
0929 03:14:28.070841 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:28.070844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:28.070897 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:28.070983 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:28.072369 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.072913 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.074418 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.074951 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.075116 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.078393 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:28.078399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:28.078402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:28.078483 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:28.079898 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:28.080454 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:28.080458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:28.080459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:28.080551 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:28.080898 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:28.080901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:28.080902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:28.081112 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:28.081200 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:28.081738 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 03:14:28.081903 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260929 03:14:28.083042 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.068677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:42909 (local address 127.20.174.67:45261)
0929 03:14:28.069613 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:28.069675 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:28.070565 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:28.071194 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:28.071202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:28.071596 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:28.072198 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.072213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.073047 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.073050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.074263 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.074274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.075212 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.075218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.075487 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.076353 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:28.076372 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:28.078215 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:28.080012 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:28.080018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:28.080032 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:28.080314 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:28.080641 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:28.080644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:28.080645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:28.080800 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:28.081211 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:28.081217 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:28.081485 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 03:14:28.081659 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 03:14:28.081998 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":557,"threads_started":1}
I20260929 03:14:28.084388 6233 heartbeater.cc:347] Connected to a master server at 127.20.174.126:42909
I20260929 03:14:28.084745 6233 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:28.085327 6233 heartbeater.cc:511] Master 127.20.174.126:42909 requested a full tablet report, sending...
I20260929 03:14:28.086580 5772 ts_manager.cc:194] Registered new tserver with Master: 1a039bad31984ebd84c8839ca4446128 (127.20.174.67:33173)
I20260929 03:14:28.087371 5772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:45261
I20260929 03:14:28.090564 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:14:28.100234 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.092599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40876 (local address 127.20.174.126:42909)
0929 03:14:28.092728 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:28.092731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:28.092880 (+ 149us) server_negotiation.cc:407] Connection header received
0929 03:14:28.092979 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:28.092981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:28.093019 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:28.093086 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:28.093838 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.094350 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.095308 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.095540 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.095619 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.097038 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:28.097041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:28.097043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:28.097066 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:28.098644 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:28.099105 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:28.099107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:28.099109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:28.099180 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:28.099435 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:28.099437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:28.099439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:28.099642 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:28.099742 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:28.100025 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 03:14:28.100116 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:14:28.103319 5772 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:14:28.106372 5772 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:14:28.121275 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.116121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.20.174.67:33173)
0929 03:14:28.116481 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:28.116496 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:28.116520 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:14:28.116601 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:28.116607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:28.116829 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:28.117363 (+ 534us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:28.117525 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.118257 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.119910 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.120613 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.120739 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.120907 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:28.121026 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 03:14:28.121121 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":136,"threads_started":1}
I20260929 03:14:28.121269 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.115296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:33173 (local address 127.0.0.1:33044)
0929 03:14:28.116029 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:28.116046 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:28.116140 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:28.117007 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:28.117015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:28.117042 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:28.117179 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.117209 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.118395 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.118401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.119792 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.119803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.120737 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.120740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.120846 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.120881 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 03:14:28.120991 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 03:14:28.121121 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260929 03:14:28.121544 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.115266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:39323 (local address 127.0.0.1:37482)
0929 03:14:28.115663 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:28.115997 (+ 334us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:28.116123 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:28.117251 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:28.117256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:28.117278 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:28.117378 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.117384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.118811 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.118814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.119983 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.119990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.121128 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.121131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.121255 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.121281 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:14:28.121329 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:14:28.121396 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":65,"thread_start_us":107,"threads_started":1}
I20260929 03:14:28.121974 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.116048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:44227 (local address 127.0.0.1:49518)
0929 03:14:28.116367 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:28.116380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:28.116461 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:28.117334 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:28.117337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:28.117353 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:28.117454 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.117463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.118960 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.118966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.120087 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.120097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.121627 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.121631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.121718 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.121743 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:14:28.121811 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:14:28.121884 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":86,"threads_started":1}
I20260929 03:14:28.122303 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.115365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37482 (local address 127.20.174.66:39323)
0929 03:14:28.116798 (+ 1433us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:28.116810 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:28.116834 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:14:28.116908 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:28.116913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:28.117098 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:28.117263 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:28.117496 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.118671 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.120100 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.120979 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.121888 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.122034 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:28.122083 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 03:14:28.122175 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1311,"thread_start_us":143,"threads_started":1}
I20260929 03:14:28.122413 6249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.116563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.20.174.65:44227)
0929 03:14:28.116936 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:28.116949 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:28.116968 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:14:28.117032 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:28.117037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:28.117188 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:28.117602 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:28.117751 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.118837 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.120213 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.121012 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.121282 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.121446 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:28.122163 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 03:14:28.122267 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260929 03:14:28.124826 6167 tablet_service.cc:1467] Processing CreateTablet for tablet a3b51eb0b00a4e45ab3c817876265a7b (DEFAULT_TABLE table=test-table [id=9ad26fc9dd4647769d0811e4e804fe9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:28.125053 6033 tablet_service.cc:1467] Processing CreateTablet for tablet a3b51eb0b00a4e45ab3c817876265a7b (DEFAULT_TABLE table=test-table [id=9ad26fc9dd4647769d0811e4e804fe9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:28.125476 5899 tablet_service.cc:1467] Processing CreateTablet for tablet a3b51eb0b00a4e45ab3c817876265a7b (DEFAULT_TABLE table=test-table [id=9ad26fc9dd4647769d0811e4e804fe9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:28.126024 6167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b51eb0b00a4e45ab3c817876265a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:28.126098 6033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b51eb0b00a4e45ab3c817876265a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:28.126365 5899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b51eb0b00a4e45ab3c817876265a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:28.132689 6250 tablet_bootstrap.cc:492] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: Bootstrap starting.
I20260929 03:14:28.133836 6251 tablet_bootstrap.cc:492] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594: Bootstrap starting.
I20260929 03:14:28.135033 6250 tablet_bootstrap.cc:654] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:28.135834 6251 tablet_bootstrap.cc:654] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:28.136466 6250 log.cc:824] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:28.136601 6251 log.cc:824] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:28.137233 6252 tablet_bootstrap.cc:492] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: Bootstrap starting.
I20260929 03:14:28.138738 6250 tablet_bootstrap.cc:492] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: No bootstrap required, opened a new log
I20260929 03:14:28.138943 6250 ts_tablet_manager.cc:1397] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:14:28.139453 6252 tablet_bootstrap.cc:654] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:28.139632 6251 tablet_bootstrap.cc:492] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594: No bootstrap required, opened a new log
I20260929 03:14:28.139838 6251 ts_tablet_manager.cc:1397] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 03:14:28.140683 6252 log.cc:824] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:28.143148 6252 tablet_bootstrap.cc:492] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: No bootstrap required, opened a new log
I20260929 03:14:28.143002 6250 raft_consensus.cc:348] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39323 } } peers { permanent_uuid: "1a039bad31984ebd84c8839ca4446128" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 33173 } } peers { permanent_uuid: "a12e5ef71af44c6ab9237ef1528e7594" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44227 } }
I20260929 03:14:28.143261 6250 raft_consensus.cc:374] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:28.143333 6252 ts_tablet_manager.cc:1397] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:14:28.143358 6250 raft_consensus.cc:729] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a039bad31984ebd84c8839ca4446128, State: Initialized, Role: FOLLOWER
I20260929 03:14:28.143738 6251 raft_consensus.cc:348] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39323 } } peers { permanent_uuid: "1a039bad31984ebd84c8839ca4446128" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 33173 } } peers { permanent_uuid: "a12e5ef71af44c6ab9237ef1528e7594" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44227 } }
I20260929 03:14:28.143831 6250 consensus_queue.cc:260] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39323 } } peers { permanent_uuid: "1a039bad31984ebd84c8839ca4446128" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 33173 } } peers { permanent_uuid: "a12e5ef71af44c6ab9237ef1528e7594" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44227 } }
I20260929 03:14:28.143978 6251 raft_consensus.cc:374] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:28.144059 6251 raft_consensus.cc:729] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12e5ef71af44c6ab9237ef1528e7594, State: Initialized, Role: FOLLOWER
I20260929 03:14:28.144469 6251 consensus_queue.cc:260] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39323 } } peers { permanent_uuid: "1a039bad31984ebd84c8839ca4446128" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 33173 } } peers { permanent_uuid: "a12e5ef71af44c6ab9237ef1528e7594" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44227 } }
I20260929 03:14:28.144846 6250 ts_tablet_manager.cc:1428] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:14:28.145140 5965 heartbeater.cc:503] Master 127.20.174.126:42909 was elected leader, sending a full tablet report...
I20260929 03:14:28.145231 6233 heartbeater.cc:503] Master 127.20.174.126:42909 was elected leader, sending a full tablet report...
I20260929 03:14:28.145437 6251 ts_tablet_manager.cc:1428] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:14:28.148584 6252 raft_consensus.cc:348] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39323 } } peers { permanent_uuid: "1a039bad31984ebd84c8839ca4446128" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 33173 } } peers { permanent_uuid: "a12e5ef71af44c6ab9237ef1528e7594" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44227 } }
I20260929 03:14:28.148988 6252 raft_consensus.cc:374] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:28.149109 6252 raft_consensus.cc:729] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b44d3f1a1254ceaa1a0ab585d0a3637, State: Initialized, Role: FOLLOWER
I20260929 03:14:28.149683 6252 consensus_queue.cc:260] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39323 } } peers { permanent_uuid: "1a039bad31984ebd84c8839ca4446128" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 33173 } } peers { permanent_uuid: "a12e5ef71af44c6ab9237ef1528e7594" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44227 } }
I20260929 03:14:28.150488 6099 heartbeater.cc:503] Master 127.20.174.126:42909 was elected leader, sending a full tablet report...
I20260929 03:14:28.150866 6252 ts_tablet_manager.cc:1428] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 03:14:28.303683 6258 raft_consensus.cc:482] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:14:28.304016 6258 raft_consensus.cc:504] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39323 } } peers { permanent_uuid: "1a039bad31984ebd84c8839ca4446128" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 33173 } } peers { permanent_uuid: "a12e5ef71af44c6ab9237ef1528e7594" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44227 } }
I20260929 03:14:28.305248 6258 leader_election.cc:288] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a039bad31984ebd84c8839ca4446128 (127.20.174.67:33173), a12e5ef71af44c6ab9237ef1528e7594 (127.20.174.65:44227)
I20260929 03:14:28.309487 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.305483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:44227 (local address 127.20.174.66:47493)
0929 03:14:28.305626 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:28.305643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:28.305762 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:28.306208 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:28.306212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:28.306250 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:28.306407 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.306414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.307263 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.307266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.308216 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.308229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.309036 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.309040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.309149 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.309179 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:14:28.309271 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:14:28.309347 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"spinlock_wait_cycles":23424}
I20260929 03:14:28.309485 6249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.305831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:47493 (local address 127.20.174.65:44227)
0929 03:14:28.305954 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:28.305958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:28.305972 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:14:28.306030 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:28.306033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:28.306081 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:28.306160 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:28.306562 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.307126 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.308355 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.308914 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.309031 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.309084 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:28.309288 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 03:14:28.309360 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:14:28.310288 5919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b51eb0b00a4e45ab3c817876265a7b" candidate_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a12e5ef71af44c6ab9237ef1528e7594" is_pre_election: true
I20260929 03:14:28.310554 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.305687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:50945 (local address 127.20.174.67:33173)
0929 03:14:28.305802 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:28.305807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:28.306280 (+ 473us) server_negotiation.cc:407] Connection header received
0929 03:14:28.306400 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:28.306403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:28.306478 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:28.306548 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:28.306929 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.307699 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.309207 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.309838 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.309930 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.309974 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:28.310316 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 03:14:28.310393 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:14:28.310693 5919 raft_consensus.cc:2454] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b44d3f1a1254ceaa1a0ab585d0a3637 in term 0.
I20260929 03:14:28.310972 6259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.305505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:33173 (local address 127.20.174.66:50945)
0929 03:14:28.306193 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:28.306206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:28.306317 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:28.306670 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:28.306673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:28.306691 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:28.306811 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.306817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.307840 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.307843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.309061 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.309072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.310067 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.310070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.310177 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.310200 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:14:28.310749 (+ 549us) client_negotiation.cc:240] Negotiation successful
0929 03:14:28.310829 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":411,"spinlock_wait_cycles":3200,"thread_start_us":106,"threads_started":1}
I20260929 03:14:28.311213 5989 leader_election.cc:302] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b44d3f1a1254ceaa1a0ab585d0a3637, a12e5ef71af44c6ab9237ef1528e7594; no voters:
I20260929 03:14:28.311604 6258 raft_consensus.cc:2781] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:14:28.311707 6258 raft_consensus.cc:482] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:14:28.311748 6258 raft_consensus.cc:3037] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:28.311735 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b51eb0b00a4e45ab3c817876265a7b" candidate_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a039bad31984ebd84c8839ca4446128" is_pre_election: true
I20260929 03:14:28.311964 6187 raft_consensus.cc:2454] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b44d3f1a1254ceaa1a0ab585d0a3637 in term 0.
I20260929 03:14:28.313045 6258 raft_consensus.cc:504] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39323 } } peers { permanent_uuid: "1a039bad31984ebd84c8839ca4446128" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 33173 } } peers { permanent_uuid: "a12e5ef71af44c6ab9237ef1528e7594" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44227 } }
I20260929 03:14:28.313413 6258 leader_election.cc:288] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [CANDIDATE]: Term 1 election: Requested vote from peers 1a039bad31984ebd84c8839ca4446128 (127.20.174.67:33173), a12e5ef71af44c6ab9237ef1528e7594 (127.20.174.65:44227)
I20260929 03:14:28.313787 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b51eb0b00a4e45ab3c817876265a7b" candidate_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a039bad31984ebd84c8839ca4446128"
I20260929 03:14:28.313805 5919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b51eb0b00a4e45ab3c817876265a7b" candidate_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a12e5ef71af44c6ab9237ef1528e7594"
I20260929 03:14:28.313923 5919 raft_consensus.cc:3037] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:28.313927 6187 raft_consensus.cc:3037] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:28.315063 5919 raft_consensus.cc:2454] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b44d3f1a1254ceaa1a0ab585d0a3637 in term 1.
I20260929 03:14:28.315070 6187 raft_consensus.cc:2454] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b44d3f1a1254ceaa1a0ab585d0a3637 in term 1.
I20260929 03:14:28.315446 5989 leader_election.cc:302] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b44d3f1a1254ceaa1a0ab585d0a3637, a12e5ef71af44c6ab9237ef1528e7594; no voters:
I20260929 03:14:28.315636 6258 raft_consensus.cc:2781] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:28.315882 6258 raft_consensus.cc:686] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 1 LEADER]: Becoming Leader. State: Replica: 1b44d3f1a1254ceaa1a0ab585d0a3637, State: Running, Role: LEADER
I20260929 03:14:28.316238 6258 consensus_queue.cc:237] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39323 } } peers { permanent_uuid: "1a039bad31984ebd84c8839ca4446128" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 33173 } } peers { permanent_uuid: "a12e5ef71af44c6ab9237ef1528e7594" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44227 } }
W20260929 03:14:28.317183 6234 tablet.cc:2369] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:14:28.319638 5774 catalog_manager.cc:5495] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b44d3f1a1254ceaa1a0ab585d0a3637 (127.20.174.66). New cstate: current_term: 1 leader_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b44d3f1a1254ceaa1a0ab585d0a3637" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 39323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a039bad31984ebd84c8839ca4446128" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a12e5ef71af44c6ab9237ef1528e7594" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44227 } health_report { overall_health: UNKNOWN } } }
W20260929 03:14:28.333024 6100 tablet.cc:2369] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:14:28.355659 5966 tablet.cc:2369] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:14:28.364315 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.360182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.20.174.66:39323)
0929 03:14:28.360303 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:28.360307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:28.360740 (+ 433us) server_negotiation.cc:407] Connection header received
0929 03:14:28.360948 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:28.360952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:28.361037 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:28.361143 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:14:28.361352 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.361847 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.362859 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.363090 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.363187 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.363516 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:14:28.363962 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:14:28.364059 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:28.364139 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 03:14:28.364194 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:14:28.370798 6187 raft_consensus.cc:1261] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128 [term 1 FOLLOWER]: Refusing update from remote peer 1b44d3f1a1254ceaa1a0ab585d0a3637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:28.370783 5919 raft_consensus.cc:1261] T a3b51eb0b00a4e45ab3c817876265a7b P a12e5ef71af44c6ab9237ef1528e7594 [term 1 FOLLOWER]: Refusing update from remote peer 1b44d3f1a1254ceaa1a0ab585d0a3637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:28.371619 6258 consensus_queue.cc:1035] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a12e5ef71af44c6ab9237ef1528e7594" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 44227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:28.371851 6260 consensus_queue.cc:1035] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a039bad31984ebd84c8839ca4446128" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:28.381402 6263 mvcc.cc:205] Tried to move back new op lower bound from 7334509233638346752 to 7334509233426059264. Current Snapshot: MvccSnapshot[applied={T|T < 7334509233638346752 or (T in {7334509233638346752})}]
I20260929 03:14:28.381669 6264 mvcc.cc:205] Tried to move back new op lower bound from 7334509233638346752 to 7334509233426059264. Current Snapshot: MvccSnapshot[applied={T|T < 7334509233638346752}]
I20260929 03:14:28.383168 6265 mvcc.cc:205] Tried to move back new op lower bound from 7334509233638346752 to 7334509233426059264. Current Snapshot: MvccSnapshot[applied={T|T < 7334509233638346752}]
I20260929 03:14:28.390235 5772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:28.390507 5772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:28.393317 5772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f3794eb755b4958ae6fcfab68475a81: Sending DeleteTablet for 3 replicas of tablet a3b51eb0b00a4e45ab3c817876265a7b
I20260929 03:14:28.394181 6033 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b51eb0b00a4e45ab3c817876265a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:28 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:37482
I20260929 03:14:28.394258 6167 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b51eb0b00a4e45ab3c817876265a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:28 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:33044
I20260929 03:14:28.394805 6272 tablet_replica.cc:330] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: stopping tablet replica
I20260929 03:14:28.394881 6273 tablet_replica.cc:330] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: stopping tablet replica
I20260929 03:14:28.395144 6273 raft_consensus.cc:2229] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:14:28.395144 6272 raft_consensus.cc:2229] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:14:28.395282 5899 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b51eb0b00a4e45ab3c817876265a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:28 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:49518
I20260929 03:14:28.395475 6273 raft_consensus.cc:2258] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:28.395539 6272 raft_consensus.cc:2258] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:28.396399 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 5836
I20260929 03:14:28.398617 6272 ts_tablet_manager.cc:1905] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:14:28.399806 6273 ts_tablet_manager.cc:1905] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:14:28.401552 6272 ts_tablet_manager.cc:1918] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:28.401666 6272 log.cc:1196] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/a3b51eb0b00a4e45ab3c817876265a7b
I20260929 03:14:28.402014 6272 ts_tablet_manager.cc:1939] T a3b51eb0b00a4e45ab3c817876265a7b P 1b44d3f1a1254ceaa1a0ab585d0a3637: Deleting consensus metadata
I20260929 03:14:28.403208 6273 ts_tablet_manager.cc:1918] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:28.403324 6273 log.cc:1196] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/a3b51eb0b00a4e45ab3c817876265a7b
I20260929 03:14:28.403736 6273 ts_tablet_manager.cc:1939] T a3b51eb0b00a4e45ab3c817876265a7b P 1a039bad31984ebd84c8839ca4446128: Deleting consensus metadata
I20260929 03:14:28.405305 5760 catalog_manager.cc:4841] TS 1a039bad31984ebd84c8839ca4446128 (127.20.174.67:33173): tablet a3b51eb0b00a4e45ab3c817876265a7b (table test-table [id=9ad26fc9dd4647769d0811e4e804fe9a]) successfully deleted
I20260929 03:14:28.406220 5758 catalog_manager.cc:4841] TS 1b44d3f1a1254ceaa1a0ab585d0a3637 (127.20.174.66:39323): tablet a3b51eb0b00a4e45ab3c817876265a7b (table test-table [id=9ad26fc9dd4647769d0811e4e804fe9a]) successfully deleted
W20260929 03:14:28.407444 5761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:44227 (error 108)
I20260929 03:14:28.408340 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 5970
I20260929 03:14:28.408635 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.408319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:44227 (local address 127.0.0.1:49534)
0929 03:14:28.408456 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:28.408536 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:44227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 03:14:28.409106 5761 catalog_manager.cc:4584] TS a12e5ef71af44c6ab9237ef1528e7594 (127.20.174.65:44227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3b51eb0b00a4e45ab3c817876265a7b: Network error: Client connection negotiation failed: client connection to 127.20.174.65:44227: connect: Connection refused (error 111)
I20260929 03:14:28.415277 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 6104
I20260929 03:14:28.421351 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 5742
2026-09-29T03:14:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2547 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T03:14:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:28Z Disabled control of system clock
I20260929 03:14:28.454140 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40899
--webserver_interface=127.20.174.126
--webserver_port=0
--builtin_ntp_servers=127.20.174.84:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:40899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:14:28.571007 6279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:28.571296 6279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:28.571334 6279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:28.575491 6279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:28.575557 6279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:28.575574 6279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:28.575591 6279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:28.575608 6279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:28.580657 6279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:40899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:28.581739 6279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:28.583330 6279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:14:28.592449 6279 server_base.cc:1034] running on GCE node
W20260929 03:14:28.592392 6287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:28.592382 6285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:28.592388 6284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:28.593415 6279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:28.594705 6279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:28.595916 6279 hybrid_clock.cc:648] HybridClock initialized: now 1790651668595901 us; error 30 us; skew 500 ppm
I20260929 03:14:28.598057 6279 webserver.cc:460] Webserver started at http://127.20.174.126:39207/ using document root <none> and password file <none>
I20260929 03:14:28.598766 6279 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:28.598837 6279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:28.599090 6279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:28.601270 6279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "4ff84bf3d04a4bb38ac6303f35665936"
format_stamp: "Formatted at 2026-09-29 03:14:28 on dist-test-slave-k1dg"
server_key: "6f442b32ffa999c46017c7379c5d3f00"
server_key_iv: "9668a5ae82d101b208a149cb442357a0"
server_key_version: "encryptionkey@0"
I20260929 03:14:28.601902 6279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "4ff84bf3d04a4bb38ac6303f35665936"
format_stamp: "Formatted at 2026-09-29 03:14:28 on dist-test-slave-k1dg"
server_key: "6f442b32ffa999c46017c7379c5d3f00"
server_key_iv: "9668a5ae82d101b208a149cb442357a0"
server_key_version: "encryptionkey@0"
I20260929 03:14:28.605938 6279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:14:28.608655 6293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:28.609990 6279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:14:28.610143 6279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "4ff84bf3d04a4bb38ac6303f35665936"
format_stamp: "Formatted at 2026-09-29 03:14:28 on dist-test-slave-k1dg"
server_key: "6f442b32ffa999c46017c7379c5d3f00"
server_key_iv: "9668a5ae82d101b208a149cb442357a0"
server_key_version: "encryptionkey@0"
I20260929 03:14:28.610267 6279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:28.631590 6279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:28.632380 6279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:28.632598 6279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:28.640290 6279 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:40899
I20260929 03:14:28.640300 6345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:40899 every 8 connection(s)
I20260929 03:14:28.641346 6279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:28.644629 6346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:28.650565 6346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936: Bootstrap starting.
I20260929 03:14:28.651304 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 6279
I20260929 03:14:28.651453 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:28.651741 21177 external_mini_cluster.cc:1428] Setting key 456e0118d583b3ee4a3ded1db677152a
I20260929 03:14:28.653844 6346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:28.654721 6346 log.cc:824] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:28.656738 6346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936: No bootstrap required, opened a new log
I20260929 03:14:28.660617 6346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff84bf3d04a4bb38ac6303f35665936" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40899 } }
I20260929 03:14:28.660962 6346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:28.661048 6346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff84bf3d04a4bb38ac6303f35665936, State: Initialized, Role: FOLLOWER
I20260929 03:14:28.661540 6346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff84bf3d04a4bb38ac6303f35665936" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40899 } }
I20260929 03:14:28.661667 6346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:28.661755 6346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:28.661820 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.653264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.20.174.126:40899)
0929 03:14:28.654002 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:28.654020 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:28.654059 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:14:28.654759 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:28.654778 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:28.655269 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:28.655636 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:28.656089 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.657135 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.658084 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.658663 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.658843 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.659768 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:28.659773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:28.659788 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:28.660207 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:28.660265 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:28.660287 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:28.660399 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:28.660564 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:14:28.660845 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":134,"threads_started":1}
I20260929 03:14:28.661885 6346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:28.662971 6346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff84bf3d04a4bb38ac6303f35665936" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40899 } }
I20260929 03:14:28.663354 6346 leader_election.cc:302] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ff84bf3d04a4bb38ac6303f35665936; no voters:
I20260929 03:14:28.663707 6346 leader_election.cc:288] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:28.663784 6351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:28.663978 6351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [term 1 LEADER]: Becoming Leader. State: Replica: 4ff84bf3d04a4bb38ac6303f35665936, State: Running, Role: LEADER
I20260929 03:14:28.664618 6351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff84bf3d04a4bb38ac6303f35665936" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40899 } }
I20260929 03:14:28.665086 6346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:28.666657 6352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ff84bf3d04a4bb38ac6303f35665936. Latest consensus state: current_term: 1 leader_uuid: "4ff84bf3d04a4bb38ac6303f35665936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff84bf3d04a4bb38ac6303f35665936" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40899 } } }
I20260929 03:14:28.666698 6353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ff84bf3d04a4bb38ac6303f35665936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ff84bf3d04a4bb38ac6303f35665936" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40899 } } }
I20260929 03:14:28.666898 6353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:28.667230 6352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:28.667554 6364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:28.671377 6364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:28.677649 6364 catalog_manager.cc:1347] Generated new cluster ID: a4b25cf1bc5941e1a71fc121cb7fa4a2
I20260929 03:14:28.677743 6364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:28.683655 6364 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:28.684696 6364 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:28.695894 6364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936: Generated new TSK 0
I20260929 03:14:28.696748 6364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:28.706218 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40899
--builtin_ntp_servers=127.20.174.84:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:14:28.824479 6370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:28.824792 6370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:28.824860 6370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:28.829128 6370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:28.829241 6370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:28.829344 6370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:14:28.834550 6370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:28.835721 6370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:28.837172 6370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:28.847664 6378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:28.847802 6370 server_base.cc:1034] running on GCE node
W20260929 03:14:28.847663 6375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:28.848026 6376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:28.848475 6370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:28.849224 6370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:28.850498 6370 hybrid_clock.cc:648] HybridClock initialized: now 1790651668850474 us; error 32 us; skew 500 ppm
I20260929 03:14:28.852536 6370 webserver.cc:460] Webserver started at http://127.20.174.65:45017/ using document root <none> and password file <none>
I20260929 03:14:28.853157 6370 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:28.853221 6370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:28.853454 6370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:28.855648 6370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73"
format_stamp: "Formatted at 2026-09-29 03:14:28 on dist-test-slave-k1dg"
server_key: "daeca4d2564621af518bfc1727948a0e"
server_key_iv: "b5304e828d4f1f7d982ac018c1aee05c"
server_key_version: "encryptionkey@0"
I20260929 03:14:28.856235 6370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73"
format_stamp: "Formatted at 2026-09-29 03:14:28 on dist-test-slave-k1dg"
server_key: "daeca4d2564621af518bfc1727948a0e"
server_key_iv: "b5304e828d4f1f7d982ac018c1aee05c"
server_key_version: "encryptionkey@0"
I20260929 03:14:28.860119 6370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 03:14:28.862864 6384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:28.864100 6370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:14:28.864252 6370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73"
format_stamp: "Formatted at 2026-09-29 03:14:28 on dist-test-slave-k1dg"
server_key: "daeca4d2564621af518bfc1727948a0e"
server_key_iv: "b5304e828d4f1f7d982ac018c1aee05c"
server_key_version: "encryptionkey@0"
I20260929 03:14:28.864406 6370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:28.876631 6370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:28.877451 6370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:28.877660 6370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:28.878312 6370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:28.879357 6370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:28.879431 6370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:28.879496 6370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:28.879549 6370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:28.889189 6370 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:39221
I20260929 03:14:28.889227 6497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:39221 every 8 connection(s)
I20260929 03:14:28.890120 6370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:14:28.893239 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 6370
I20260929 03:14:28.893373 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:14:28.893712 21177 external_mini_cluster.cc:1428] Setting key f0c68ef87c6c0b857ba1d63d0dbea024
I20260929 03:14:28.896104 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40899
--builtin_ntp_servers=127.20.174.84:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:14:28.901976 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.892013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:47825 (local address 127.20.174.126:40899)
0929 03:14:28.892191 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:28.892196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:28.893434 (+ 1238us) server_negotiation.cc:407] Connection header received
0929 03:14:28.895458 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:28.895461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:28.895517 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:28.895607 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:28.896965 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.897783 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.899560 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.899996 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.900135 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.901343 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:28.901350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:28.901356 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:28.901379 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:28.901412 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:28.901426 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:28.901514 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:28.901665 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 03:14:28.901748 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 03:14:28.902956 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:28.892452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40899 (local address 127.20.174.65:47825)
0929 03:14:28.893215 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:28.893257 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:28.895164 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:28.895785 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:28.895792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:28.895892 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:28.896677 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.896713 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.897925 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.897931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.899399 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:28.899411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:28.900173 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:28.900176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:28.900332 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:28.901039 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:14:28.901040 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:28.901087 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:14:28.901162 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:28.901518 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:28.901525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:28.901529 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:14:28.901634 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:14:28.901889 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260929 03:14:28.904300 6498 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40899
I20260929 03:14:28.904561 6498 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:28.905120 6498 heartbeater.cc:511] Master 127.20.174.126:40899 requested a full tablet report, sending...
I20260929 03:14:28.906772 6310 ts_manager.cc:194] Registered new tserver with Master: b1b6ba0124ef42eaaeb2798fcc5bcf73 (127.20.174.65:39221)
I20260929 03:14:28.907969 6310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.65:47825
W20260929 03:14:29.013842 6501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:29.014199 6501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:29.014277 6501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:29.018605 6501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:29.018709 6501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:29.018810 6501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:14:29.023810 6501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:29.024938 6501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:29.026461 6501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:29.036304 6509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:29.036358 6501 server_base.cc:1034] running on GCE node
W20260929 03:14:29.036307 6507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:29.036307 6506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:29.036921 6501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:29.037616 6501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:29.038812 6501 hybrid_clock.cc:648] HybridClock initialized: now 1790651669038781 us; error 49 us; skew 500 ppm
I20260929 03:14:29.040863 6501 webserver.cc:460] Webserver started at http://127.20.174.66:43861/ using document root <none> and password file <none>
I20260929 03:14:29.041493 6501 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:29.041594 6501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:29.041873 6501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:29.044129 6501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "29d563c244f24876ab2e14177e44fe3d"
format_stamp: "Formatted at 2026-09-29 03:14:29 on dist-test-slave-k1dg"
server_key: "d0d0a0a3028d55fe884bff74e8328532"
server_key_iv: "3c49ccea323760f8492d2a1eebf6981b"
server_key_version: "encryptionkey@0"
I20260929 03:14:29.044705 6501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "29d563c244f24876ab2e14177e44fe3d"
format_stamp: "Formatted at 2026-09-29 03:14:29 on dist-test-slave-k1dg"
server_key: "d0d0a0a3028d55fe884bff74e8328532"
server_key_iv: "3c49ccea323760f8492d2a1eebf6981b"
server_key_version: "encryptionkey@0"
I20260929 03:14:29.048491 6501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 03:14:29.051046 6515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:29.052364 6501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:14:29.052542 6501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "29d563c244f24876ab2e14177e44fe3d"
format_stamp: "Formatted at 2026-09-29 03:14:29 on dist-test-slave-k1dg"
server_key: "d0d0a0a3028d55fe884bff74e8328532"
server_key_iv: "3c49ccea323760f8492d2a1eebf6981b"
server_key_version: "encryptionkey@0"
I20260929 03:14:29.052661 6501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:29.071862 6501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:29.072631 6501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:29.072842 6501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:29.073508 6501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:29.074569 6501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:29.074638 6501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:29.074710 6501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:29.074761 6501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:29.084044 6501 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:41733
I20260929 03:14:29.084076 6628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:41733 every 8 connection(s)
I20260929 03:14:29.084992 6501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:14:29.094046 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 6501
I20260929 03:14:29.094204 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:14:29.094636 21177 external_mini_cluster.cc:1428] Setting key fafa8a8928a77fd4a261d55ec218af18
I20260929 03:14:29.094937 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.087049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:42223 (local address 127.20.174.126:40899)
0929 03:14:29.087221 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:29.087226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:29.088363 (+ 1137us) server_negotiation.cc:407] Connection header received
0929 03:14:29.089753 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:29.089756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:29.089804 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:29.089885 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:29.090833 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.091486 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.092634 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.092843 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.092964 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.094192 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:29.094196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:29.094199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:29.094217 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:29.094243 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:29.094258 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:29.094362 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:29.094638 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 03:14:29.094730 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:14:29.095882 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.087494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40899 (local address 127.20.174.66:42223)
0929 03:14:29.088194 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:29.088243 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:29.089548 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:29.090052 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:29.090059 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:29.090144 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:29.090702 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.090715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.091612 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.091615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.092524 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.092533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.093023 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.093025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.093173 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.093843 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:14:29.093845 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:29.093890 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:14:29.093960 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:29.094418 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:29.094454 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:29.094461 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:14:29.094612 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 03:14:29.094933 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":128,"threads_started":1}
I20260929 03:14:29.097045 6629 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40899
I20260929 03:14:29.097059 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40899
--builtin_ntp_servers=127.20.174.84:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:14:29.097355 6629 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:29.098031 6629 heartbeater.cc:511] Master 127.20.174.126:40899 requested a full tablet report, sending...
I20260929 03:14:29.099381 6310 ts_manager.cc:194] Registered new tserver with Master: 29d563c244f24876ab2e14177e44fe3d (127.20.174.66:41733)
I20260929 03:14:29.100353 6310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.66:42223
W20260929 03:14:29.221182 6632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:29.221484 6632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:29.221525 6632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:29.225776 6632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:29.225847 6632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:29.225906 6632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:14:29.230805 6632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:29.231906 6632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:29.233562 6632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:14:29.243911 6632 server_base.cc:1034] running on GCE node
W20260929 03:14:29.243921 6638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:29.243835 6640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:29.244154 6637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:29.244592 6632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:29.245317 6632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:29.246503 6632 hybrid_clock.cc:648] HybridClock initialized: now 1790651669246459 us; error 59 us; skew 500 ppm
I20260929 03:14:29.248538 6632 webserver.cc:460] Webserver started at http://127.20.174.67:37043/ using document root <none> and password file <none>
I20260929 03:14:29.249147 6632 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:29.249212 6632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:29.249433 6632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:29.251631 6632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "f2d41e4d4f574359a073627f5590126c"
format_stamp: "Formatted at 2026-09-29 03:14:29 on dist-test-slave-k1dg"
server_key: "219d13e5b108e3a6feee0855e4a085ef"
server_key_iv: "26045ee4e8c4226b9deb1d8a8f778a1f"
server_key_version: "encryptionkey@0"
I20260929 03:14:29.252225 6632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "f2d41e4d4f574359a073627f5590126c"
format_stamp: "Formatted at 2026-09-29 03:14:29 on dist-test-slave-k1dg"
server_key: "219d13e5b108e3a6feee0855e4a085ef"
server_key_iv: "26045ee4e8c4226b9deb1d8a8f778a1f"
server_key_version: "encryptionkey@0"
I20260929 03:14:29.256305 6632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:14:29.259011 6646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:29.260365 6632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:29.260524 6632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "f2d41e4d4f574359a073627f5590126c"
format_stamp: "Formatted at 2026-09-29 03:14:29 on dist-test-slave-k1dg"
server_key: "219d13e5b108e3a6feee0855e4a085ef"
server_key_iv: "26045ee4e8c4226b9deb1d8a8f778a1f"
server_key_version: "encryptionkey@0"
I20260929 03:14:29.260656 6632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:29.290314 6632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:29.291147 6632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:29.291381 6632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:29.292136 6632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:29.293174 6632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:29.293257 6632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:29.293335 6632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:29.293390 6632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:29.303036 6632 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:39413
I20260929 03:14:29.303056 6759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:39413 every 8 connection(s)
I20260929 03:14:29.304044 6632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:14:29.305352 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 6632
I20260929 03:14:29.305469 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:14:29.305780 21177 external_mini_cluster.cc:1428] Setting key 0bb739cf9b22c98cd4c4227fce8aafc5
I20260929 03:14:29.314675 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.306313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:49267 (local address 127.20.174.126:40899)
0929 03:14:29.306506 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:29.306512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:29.307858 (+ 1346us) server_negotiation.cc:407] Connection header received
0929 03:14:29.309408 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:29.309411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:29.309457 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:29.309528 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:29.310756 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.311309 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.312594 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.312818 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.312967 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.314142 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:29.314146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:29.314148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:29.314161 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:29.314179 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:29.314187 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:29.314256 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:29.314506 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 03:14:29.314559 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:14:29.315734 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.306873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40899 (local address 127.20.174.67:49267)
0929 03:14:29.307689 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:29.307753 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:29.309192 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:29.309707 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:29.309811 (+ 104us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:29.309907 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:29.310585 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.310602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.311432 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.311435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.312469 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.312479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.312985 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.312988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.313139 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.313861 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:14:29.313863 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:29.313905 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:14:29.313970 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:29.314305 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:29.314314 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:29.314319 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:14:29.314497 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 03:14:29.314818 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":149,"threads_started":1}
I20260929 03:14:29.316984 6760 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40899
I20260929 03:14:29.317207 6760 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:29.317731 6760 heartbeater.cc:511] Master 127.20.174.126:40899 requested a full tablet report, sending...
I20260929 03:14:29.318986 6310 ts_manager.cc:194] Registered new tserver with Master: f2d41e4d4f574359a073627f5590126c (127.20.174.67:39413)
I20260929 03:14:29.319694 6310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.67:49267
I20260929 03:14:29.319958 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:14:29.326046 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.322134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59150 (local address 127.20.174.126:40899)
0929 03:14:29.322288 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:29.322292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:29.322523 (+ 231us) server_negotiation.cc:407] Connection header received
0929 03:14:29.322667 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:29.322670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:29.322716 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:29.322792 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:29.323080 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.323601 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.324451 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.324687 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.324796 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.325596 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:29.325602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:29.325610 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:29.325622 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:29.325640 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:29.325647 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:29.325732 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:29.325826 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 03:14:29.325880 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:14:29.329236 6310 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:14:29.332437 6310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:14:29.347741 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.341661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:39413 (local address 127.0.0.1:45008)
0929 03:14:29.342322 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:29.342336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:29.342474 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:29.343219 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:29.343222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:29.343249 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:29.343356 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.343362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.344838 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.344841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.346092 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.346101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.347208 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.347214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.347346 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.347388 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:14:29.347499 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 03:14:29.347599 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":53,"thread_start_us":82,"threads_started":1}
I20260929 03:14:29.347898 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.342063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.20.174.67:39413)
0929 03:14:29.342723 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:29.342733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:29.342752 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:14:29.342816 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:29.342821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:29.343071 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:29.343522 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:29.343654 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.344718 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.346218 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.347042 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.347333 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.347529 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:29.347586 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 03:14:29.347710 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":170,"threads_started":1}
I20260929 03:14:29.348044 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.341274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:41733 (local address 127.0.0.1:46590)
0929 03:14:29.341974 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:29.342026 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:29.342160 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:29.343057 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:29.343065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:29.343090 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:29.343233 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.343239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.344654 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.344657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.346328 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.346339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.347592 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.347596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.347706 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.347730 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:14:29.347814 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:14:29.347893 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":161,"thread_start_us":77,"threads_started":1}
I20260929 03:14:29.348690 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.341459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.20.174.66:41733)
0929 03:14:29.342501 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:29.342516 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:29.342543 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:14:29.342624 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:29.342629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:29.342840 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:29.343052 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:29.343359 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.344567 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.346497 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.347259 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.347356 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.347529 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:29.348435 (+ 906us) server_negotiation.cc:299] Negotiation successful
0929 03:14:29.348541 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":122,"threads_started":1}
I20260929 03:14:29.350790 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.341460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:39221 (local address 127.0.0.1:50408)
0929 03:14:29.342164 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:29.342179 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:29.342275 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:29.343494 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:29.343497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:29.343514 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:29.343594 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.343602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.344607 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.344613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.346328 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.346339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.347171 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.347174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.347986 (+ 812us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.348008 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:14:29.350560 (+ 2552us) client_negotiation.cc:240] Negotiation successful
0929 03:14:29.350647 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":294,"thread_start_us":110,"threads_started":1}
I20260929 03:14:29.350894 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.342590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.20.174.65:39221)
0929 03:14:29.343056 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:29.343067 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:29.343088 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:14:29.343158 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:29.343163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:29.343352 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:29.343505 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:29.343689 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.344493 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.346494 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.347036 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.347172 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.347411 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:29.350608 (+ 3197us) server_negotiation.cc:299] Negotiation successful
0929 03:14:29.350723 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":114,"threads_started":1}
I20260929 03:14:29.351263 6694 tablet_service.cc:1467] Processing CreateTablet for tablet 16052f2c2af3467caa15700391ca2edb (DEFAULT_TABLE table=test-table [id=28d1f4a878f341c9bce9e9f719b91b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:29.351297 6563 tablet_service.cc:1467] Processing CreateTablet for tablet 16052f2c2af3467caa15700391ca2edb (DEFAULT_TABLE table=test-table [id=28d1f4a878f341c9bce9e9f719b91b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:29.352468 6563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16052f2c2af3467caa15700391ca2edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:29.352468 6694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16052f2c2af3467caa15700391ca2edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:29.353675 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 16052f2c2af3467caa15700391ca2edb (DEFAULT_TABLE table=test-table [id=28d1f4a878f341c9bce9e9f719b91b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:29.354569 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16052f2c2af3467caa15700391ca2edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:29.360041 6777 tablet_bootstrap.cc:492] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: Bootstrap starting.
I20260929 03:14:29.360505 6778 tablet_bootstrap.cc:492] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: Bootstrap starting.
I20260929 03:14:29.361317 6779 tablet_bootstrap.cc:492] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73: Bootstrap starting.
I20260929 03:14:29.363690 6779 tablet_bootstrap.cc:654] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:29.364667 6779 log.cc:824] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:29.365123 6778 tablet_bootstrap.cc:654] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:29.365865 6777 tablet_bootstrap.cc:654] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:29.366714 6777 log.cc:824] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:29.366837 6778 log.cc:824] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:29.367058 6779 tablet_bootstrap.cc:492] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73: No bootstrap required, opened a new log
I20260929 03:14:29.367348 6779 ts_tablet_manager.cc:1397] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:14:29.368914 6777 tablet_bootstrap.cc:492] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: No bootstrap required, opened a new log
I20260929 03:14:29.369104 6777 ts_tablet_manager.cc:1397] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 03:14:29.371701 6778 tablet_bootstrap.cc:492] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: No bootstrap required, opened a new log
I20260929 03:14:29.371927 6778 ts_tablet_manager.cc:1397] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260929 03:14:29.371874 6779 raft_consensus.cc:348] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d563c244f24876ab2e14177e44fe3d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41733 } } peers { permanent_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 39221 } } peers { permanent_uuid: "f2d41e4d4f574359a073627f5590126c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39413 } }
I20260929 03:14:29.372169 6779 raft_consensus.cc:374] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:29.372253 6779 raft_consensus.cc:729] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b6ba0124ef42eaaeb2798fcc5bcf73, State: Initialized, Role: FOLLOWER
I20260929 03:14:29.372730 6779 consensus_queue.cc:260] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d563c244f24876ab2e14177e44fe3d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41733 } } peers { permanent_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 39221 } } peers { permanent_uuid: "f2d41e4d4f574359a073627f5590126c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39413 } }
I20260929 03:14:29.373471 6498 heartbeater.cc:503] Master 127.20.174.126:40899 was elected leader, sending a full tablet report...
I20260929 03:14:29.373400 6777 raft_consensus.cc:348] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d563c244f24876ab2e14177e44fe3d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41733 } } peers { permanent_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 39221 } } peers { permanent_uuid: "f2d41e4d4f574359a073627f5590126c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39413 } }
I20260929 03:14:29.373641 6777 raft_consensus.cc:374] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:29.373695 6777 raft_consensus.cc:729] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d563c244f24876ab2e14177e44fe3d, State: Initialized, Role: FOLLOWER
I20260929 03:14:29.373835 6779 ts_tablet_manager.cc:1428] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:14:29.374142 6777 consensus_queue.cc:260] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d563c244f24876ab2e14177e44fe3d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41733 } } peers { permanent_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 39221 } } peers { permanent_uuid: "f2d41e4d4f574359a073627f5590126c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39413 } }
I20260929 03:14:29.374944 6629 heartbeater.cc:503] Master 127.20.174.126:40899 was elected leader, sending a full tablet report...
I20260929 03:14:29.375069 6777 ts_tablet_manager.cc:1428] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:14:29.376034 6778 raft_consensus.cc:348] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d563c244f24876ab2e14177e44fe3d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41733 } } peers { permanent_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 39221 } } peers { permanent_uuid: "f2d41e4d4f574359a073627f5590126c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39413 } }
I20260929 03:14:29.376322 6778 raft_consensus.cc:374] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:29.376415 6778 raft_consensus.cc:729] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d41e4d4f574359a073627f5590126c, State: Initialized, Role: FOLLOWER
I20260929 03:14:29.376880 6778 consensus_queue.cc:260] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d563c244f24876ab2e14177e44fe3d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41733 } } peers { permanent_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 39221 } } peers { permanent_uuid: "f2d41e4d4f574359a073627f5590126c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39413 } }
I20260929 03:14:29.377581 6760 heartbeater.cc:503] Master 127.20.174.126:40899 was elected leader, sending a full tablet report...
I20260929 03:14:29.377925 6778 ts_tablet_manager.cc:1428] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 03:14:29.391384 6499 tablet.cc:2369] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:14:29.547233 6784 raft_consensus.cc:482] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:14:29.547518 6784 raft_consensus.cc:504] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d563c244f24876ab2e14177e44fe3d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41733 } } peers { permanent_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 39221 } } peers { permanent_uuid: "f2d41e4d4f574359a073627f5590126c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39413 } }
I20260929 03:14:29.548727 6784 leader_election.cc:288] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1b6ba0124ef42eaaeb2798fcc5bcf73 (127.20.174.65:39221), f2d41e4d4f574359a073627f5590126c (127.20.174.67:39413)
I20260929 03:14:29.552896 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.549181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:34147 (local address 127.20.174.65:39221)
0929 03:14:29.549308 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:29.549312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:29.549643 (+ 331us) server_negotiation.cc:407] Connection header received
0929 03:14:29.549775 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:29.549778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:29.549841 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:29.549920 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:29.550195 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.550768 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.551924 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.552315 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.552454 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.552507 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:29.552674 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 03:14:29.552753 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:14:29.552904 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.548994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:39413 (local address 127.20.174.66:56963)
0929 03:14:29.549174 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:29.549191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:29.549293 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:29.549568 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:29.549570 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:29.549585 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:29.549774 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.549781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.550817 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.550821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.551738 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.551747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.552468 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.552472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.552578 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.552594 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:14:29.552694 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 03:14:29.552760 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":54,"spinlock_wait_cycles":9472}
I20260929 03:14:29.552917 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.548997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:39221 (local address 127.20.174.66:34147)
0929 03:14:29.549540 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:29.549554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:29.549658 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:29.549945 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:29.549948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:29.549966 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:29.550080 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.550086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.550900 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.550903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.551749 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:29.551759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.552439 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.552443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:29.552541 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.552567 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:14:29.552629 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:14:29.552697 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":151,"thread_start_us":120,"threads_started":1}
I20260929 03:14:29.552985 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.549177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:56963 (local address 127.20.174.67:39413)
0929 03:14:29.549307 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:29.549312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:29.549329 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:14:29.549397 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:29.549401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:29.549452 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:29.549547 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:29.549928 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.550666 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.551878 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.552319 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.552455 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.552508 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:29.552788 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 03:14:29.552865 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:14:29.553571 6714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16052f2c2af3467caa15700391ca2edb" candidate_uuid: "29d563c244f24876ab2e14177e44fe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d41e4d4f574359a073627f5590126c" is_pre_election: true
I20260929 03:14:29.553638 6452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16052f2c2af3467caa15700391ca2edb" candidate_uuid: "29d563c244f24876ab2e14177e44fe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" is_pre_election: true
I20260929 03:14:29.553903 6714 raft_consensus.cc:2454] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d563c244f24876ab2e14177e44fe3d in term 0.
I20260929 03:14:29.553943 6452 raft_consensus.cc:2454] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d563c244f24876ab2e14177e44fe3d in term 0.
I20260929 03:14:29.554587 6519 leader_election.cc:302] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29d563c244f24876ab2e14177e44fe3d, b1b6ba0124ef42eaaeb2798fcc5bcf73, f2d41e4d4f574359a073627f5590126c; no voters:
I20260929 03:14:29.554978 6784 raft_consensus.cc:2781] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:14:29.555088 6784 raft_consensus.cc:482] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 03:14:29.555109 6761 tablet.cc:2369] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:14:29.555143 6784 raft_consensus.cc:3037] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:29.556248 6784 raft_consensus.cc:504] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d563c244f24876ab2e14177e44fe3d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41733 } } peers { permanent_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 39221 } } peers { permanent_uuid: "f2d41e4d4f574359a073627f5590126c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39413 } }
I20260929 03:14:29.556581 6784 leader_election.cc:288] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [CANDIDATE]: Term 1 election: Requested vote from peers b1b6ba0124ef42eaaeb2798fcc5bcf73 (127.20.174.65:39221), f2d41e4d4f574359a073627f5590126c (127.20.174.67:39413)
I20260929 03:14:29.556885 6452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16052f2c2af3467caa15700391ca2edb" candidate_uuid: "29d563c244f24876ab2e14177e44fe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73"
I20260929 03:14:29.557004 6714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16052f2c2af3467caa15700391ca2edb" candidate_uuid: "29d563c244f24876ab2e14177e44fe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d41e4d4f574359a073627f5590126c"
I20260929 03:14:29.557032 6452 raft_consensus.cc:3037] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:29.557166 6714 raft_consensus.cc:3037] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:29.558194 6452 raft_consensus.cc:2454] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d563c244f24876ab2e14177e44fe3d in term 1.
I20260929 03:14:29.558256 6714 raft_consensus.cc:2454] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d563c244f24876ab2e14177e44fe3d in term 1.
I20260929 03:14:29.558559 6517 leader_election.cc:302] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29d563c244f24876ab2e14177e44fe3d, b1b6ba0124ef42eaaeb2798fcc5bcf73, f2d41e4d4f574359a073627f5590126c; no voters:
I20260929 03:14:29.558776 6784 raft_consensus.cc:2781] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:29.558943 6784 raft_consensus.cc:686] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 1 LEADER]: Becoming Leader. State: Replica: 29d563c244f24876ab2e14177e44fe3d, State: Running, Role: LEADER
I20260929 03:14:29.559268 6784 consensus_queue.cc:237] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d563c244f24876ab2e14177e44fe3d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41733 } } peers { permanent_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 39221 } } peers { permanent_uuid: "f2d41e4d4f574359a073627f5590126c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39413 } }
I20260929 03:14:29.562368 6310 catalog_manager.cc:5495] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d563c244f24876ab2e14177e44fe3d (127.20.174.66). New cstate: current_term: 1 leader_uuid: "29d563c244f24876ab2e14177e44fe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d563c244f24876ab2e14177e44fe3d" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 39221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2d41e4d4f574359a073627f5590126c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39413 } health_report { overall_health: UNKNOWN } } }
I20260929 03:14:29.584647 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.581037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46606 (local address 127.20.174.66:41733)
0929 03:14:29.581209 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:29.581213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:29.581228 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:14:29.581308 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:29.581310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:29.581382 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:29.581487 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:14:29.581774 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.582342 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.583242 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:29.583477 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:29.583568 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:29.583858 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:14:29.584319 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:14:29.584420 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:29.584479 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 03:14:29.584530 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 03:14:29.586190 6630 tablet.cc:2369] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:14:29.591143 6714 raft_consensus.cc:1261] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c [term 1 FOLLOWER]: Refusing update from remote peer 29d563c244f24876ab2e14177e44fe3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:29.591161 6452 raft_consensus.cc:1261] T 16052f2c2af3467caa15700391ca2edb P b1b6ba0124ef42eaaeb2798fcc5bcf73 [term 1 FOLLOWER]: Refusing update from remote peer 29d563c244f24876ab2e14177e44fe3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:29.591929 6787 consensus_queue.cc:1035] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d41e4d4f574359a073627f5590126c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:29.592134 6784 consensus_queue.cc:1035] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1b6ba0124ef42eaaeb2798fcc5bcf73" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 39221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:29.599149 6790 mvcc.cc:205] Tried to move back new op lower bound from 7334509238637383680 to 7334509238517657600. Current Snapshot: MvccSnapshot[applied={T|T < 7334509238637383680}]
I20260929 03:14:29.601328 6791 mvcc.cc:205] Tried to move back new op lower bound from 7334509238637383680 to 7334509238517657600. Current Snapshot: MvccSnapshot[applied={T|T < 7334509238637383680}]
I20260929 03:14:29.602830 6792 mvcc.cc:205] Tried to move back new op lower bound from 7334509238637383680 to 7334509238517657600. Current Snapshot: MvccSnapshot[applied={T|T < 7334509238637383680}]
I20260929 03:14:29.608248 6309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:29.608438 6309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:29.611047 6309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ff84bf3d04a4bb38ac6303f35665936: Sending DeleteTablet for 3 replicas of tablet 16052f2c2af3467caa15700391ca2edb
I20260929 03:14:29.611824 6563 tablet_service.cc:1514] Processing DeleteTablet for tablet 16052f2c2af3467caa15700391ca2edb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:29 UTC) from {username='slave'} at 127.0.0.1:46590
I20260929 03:14:29.611898 6426 tablet_service.cc:1514] Processing DeleteTablet for tablet 16052f2c2af3467caa15700391ca2edb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:29 UTC) from {username='slave'} at 127.0.0.1:50408
I20260929 03:14:29.612349 6800 tablet_replica.cc:330] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: stopping tablet replica
I20260929 03:14:29.612383 6694 tablet_service.cc:1514] Processing DeleteTablet for tablet 16052f2c2af3467caa15700391ca2edb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:29 UTC) from {username='slave'} at 127.0.0.1:45008
I20260929 03:14:29.612686 6800 raft_consensus.cc:2229] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:14:29.612836 6802 tablet_replica.cc:330] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: stopping tablet replica
I20260929 03:14:29.613169 6802 raft_consensus.cc:2229] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:14:29.613221 6800 raft_consensus.cc:2258] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:29.613511 6802 raft_consensus.cc:2258] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:29.613629 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 6370
I20260929 03:14:29.615434 6800 ts_tablet_manager.cc:1905] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:14:29.616289 6802 ts_tablet_manager.cc:1905] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:14:29.618646 6800 ts_tablet_manager.cc:1918] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:29.618829 6800 log.cc:1196] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/16052f2c2af3467caa15700391ca2edb
I20260929 03:14:29.619227 6800 ts_tablet_manager.cc:1939] T 16052f2c2af3467caa15700391ca2edb P 29d563c244f24876ab2e14177e44fe3d: Deleting consensus metadata
I20260929 03:14:29.619283 6802 ts_tablet_manager.cc:1918] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:29.619453 6802 log.cc:1196] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/16052f2c2af3467caa15700391ca2edb
I20260929 03:14:29.619805 6802 ts_tablet_manager.cc:1939] T 16052f2c2af3467caa15700391ca2edb P f2d41e4d4f574359a073627f5590126c: Deleting consensus metadata
I20260929 03:14:29.620568 6294 catalog_manager.cc:4841] TS 29d563c244f24876ab2e14177e44fe3d (127.20.174.66:41733): tablet 16052f2c2af3467caa15700391ca2edb (table test-table [id=28d1f4a878f341c9bce9e9f719b91b86]) successfully deleted
I20260929 03:14:29.621029 6295 catalog_manager.cc:4841] TS f2d41e4d4f574359a073627f5590126c (127.20.174.67:39413): tablet 16052f2c2af3467caa15700391ca2edb (table test-table [id=28d1f4a878f341c9bce9e9f719b91b86]) successfully deleted
W20260929 03:14:29.622023 6297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:39221 (error 108)
I20260929 03:14:29.622857 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 6501
I20260929 03:14:29.623118 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:29.622815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:39221 (local address 127.0.0.1:50422)
0929 03:14:29.622949 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:29.623024 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:39221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 03:14:29.626945 6297 catalog_manager.cc:4584] TS b1b6ba0124ef42eaaeb2798fcc5bcf73 (127.20.174.65:39221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 16052f2c2af3467caa15700391ca2edb: Network error: Client connection negotiation failed: client connection to 127.20.174.65:39221: connect: Connection refused (error 111)
I20260929 03:14:29.628883 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 6632
I20260929 03:14:29.634687 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 6279
2026-09-29T03:14:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1211 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:29 dist-test-slave-k1dg krb5kdc[6806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:29 dist-test-slave-k1dg krb5kdc[6806](info): set up 2 sockets
Sep 29 03:14:29 dist-test-slave-k1dg krb5kdc[6806](info): commencing operation
krb5kdc: starting...
W20260929 03:14:30.702824 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:30 dist-test-slave-k1dg krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:14:30.842715 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:38107
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:38107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:30.963424 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:30.963716 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:30.963758 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:30.967793 6819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:30.967857 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:30.967876 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:30.967895 6819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:30.967909 6819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:30.973336 6819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:38107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:30.974449 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:30.975451 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:30.985165 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:30.985149 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:30.985258 6819 server_base.cc:1034] running on GCE node
W20260929 03:14:30.985149 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:30.986172 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:30.987502 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:30.988690 6819 hybrid_clock.cc:648] HybridClock initialized: now 1790651670988653 us; error 56 us; skew 500 ppm
Sep 29 03:14:30 dist-test-slave-k1dg krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651670, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:30.996212 6819 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:14:30.997439 6819 webserver.cc:460] Webserver started at http://127.20.174.126:38641/ using document root <none> and password file <none>
I20260929 03:14:30.998086 6819 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:30.998154 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:30.998405 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:31.000746 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "f4ed6c22267b40228b6b6bb32726063d"
format_stamp: "Formatted at 2026-09-29 03:14:30 on dist-test-slave-k1dg"
server_key: "e395c2da9a6a787206b138e754e9ed6e"
server_key_iv: "8b7887e7c312b2cf009e92fb1bd90d56"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.001365 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "f4ed6c22267b40228b6b6bb32726063d"
format_stamp: "Formatted at 2026-09-29 03:14:30 on dist-test-slave-k1dg"
server_key: "e395c2da9a6a787206b138e754e9ed6e"
server_key_iv: "8b7887e7c312b2cf009e92fb1bd90d56"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.005360 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 03:14:31.007936 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:31.009392 6819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:14:31.009608 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "f4ed6c22267b40228b6b6bb32726063d"
format_stamp: "Formatted at 2026-09-29 03:14:30 on dist-test-slave-k1dg"
server_key: "e395c2da9a6a787206b138e754e9ed6e"
server_key_iv: "8b7887e7c312b2cf009e92fb1bd90d56"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.009725 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:31.032567 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:31.036266 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:31.036535 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:31.044896 6819 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:38107
I20260929 03:14:31.044947 6886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:38107 every 8 connection(s)
I20260929 03:14:31.046034 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:31.049086 6887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:31.050288 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 6819
I20260929 03:14:31.050501 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:31.050827 21177 external_mini_cluster.cc:1428] Setting key c9bfe8f0b04052582c9b12cd7ec3c744
I20260929 03:14:31.055932 6887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d: Bootstrap starting.
I20260929 03:14:31.058938 6887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:31.059999 6887 log.cc:824] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:14:31 dist-test-slave-k1dg krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:14:31.062613 6887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d: No bootstrap required, opened a new log
I20260929 03:14:31.066855 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.052479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.20.174.126:38107)
0929 03:14:31.052926 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:31.052942 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:31.052978 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:14:31.053623 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:31.053640 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:31.054034 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:31.054341 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:31.055341 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.056407 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.057613 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.058098 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.058228 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.061456 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:31.061462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:31.061479 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:31.061508 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:31.063482 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:31.064044 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:31.064049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:31.064059 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:31.064185 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:31.064535 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:31.064537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:31.064538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:31.064995 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:31.065169 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:31.065521 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 03:14:31.065871 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":155,"threads_started":1}
I20260929 03:14:31.067075 6887 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ed6c22267b40228b6b6bb32726063d" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38107 } }
I20260929 03:14:31.067387 6887 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:31.067451 6887 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ed6c22267b40228b6b6bb32726063d, State: Initialized, Role: FOLLOWER
I20260929 03:14:31.067968 6887 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ed6c22267b40228b6b6bb32726063d" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38107 } }
I20260929 03:14:31.068171 6887 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:31.068261 6887 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:31.068382 6887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:31.069454 6887 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ed6c22267b40228b6b6bb32726063d" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38107 } }
I20260929 03:14:31.069809 6887 leader_election.cc:302] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4ed6c22267b40228b6b6bb32726063d; no voters:
I20260929 03:14:31.070160 6887 leader_election.cc:288] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:31.070358 6892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:31.070626 6892 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [term 1 LEADER]: Becoming Leader. State: Replica: f4ed6c22267b40228b6b6bb32726063d, State: Running, Role: LEADER
I20260929 03:14:31.071071 6892 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ed6c22267b40228b6b6bb32726063d" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38107 } }
I20260929 03:14:31.071573 6887 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:31.072844 6893 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4ed6c22267b40228b6b6bb32726063d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ed6c22267b40228b6b6bb32726063d" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38107 } } }
I20260929 03:14:31.072981 6893 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:31.073395 6894 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4ed6c22267b40228b6b6bb32726063d. Latest consensus state: current_term: 1 leader_uuid: "f4ed6c22267b40228b6b6bb32726063d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ed6c22267b40228b6b6bb32726063d" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 38107 } } }
I20260929 03:14:31.073577 6894 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d [sys.catalog]: This master's current role is: LEADER
W20260929 03:14:31.076781 6907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:14:31.076861 6907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:14:31.076956 6901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:31.077701 6901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:31.083616 6901 catalog_manager.cc:1347] Generated new cluster ID: 395015cfc20847ad975e1e74a1229a25
I20260929 03:14:31.083706 6901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:31.092463 6901 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:31.093950 6901 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:31.103276 6901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d: Generated new TSK 0
I20260929 03:14:31.104048 6901 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:14:31.143097 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38107
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:31.264266 6913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:31.264587 6913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:31.264634 6913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:31.269063 6913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:31.269155 6913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:31.269222 6913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:14:31.275362 6913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:31.276633 6913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:31.277781 6913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:31.288941 6919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:31.288964 6921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:31.289279 6913 server_base.cc:1034] running on GCE node
W20260929 03:14:31.289109 6918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:31.289875 6913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:31.290728 6913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:31.291957 6913 hybrid_clock.cc:648] HybridClock initialized: now 1790651671291918 us; error 61 us; skew 500 ppm
Sep 29 03:14:31 dist-test-slave-k1dg krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:31.295260 6913 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:14:31.296375 6913 webserver.cc:460] Webserver started at http://127.20.174.65:45093/ using document root <none> and password file <none>
I20260929 03:14:31.297083 6913 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:31.297210 6913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:31.297444 6913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:31.299778 6913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "b1a5a7a340224b9aa0caf9125e713120"
format_stamp: "Formatted at 2026-09-29 03:14:31 on dist-test-slave-k1dg"
server_key: "295996b12fc46de25fa0037409cd2c93"
server_key_iv: "8a065174df4d32e74c5c59af799f39d3"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.300374 6913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "b1a5a7a340224b9aa0caf9125e713120"
format_stamp: "Formatted at 2026-09-29 03:14:31 on dist-test-slave-k1dg"
server_key: "295996b12fc46de25fa0037409cd2c93"
server_key_iv: "8a065174df4d32e74c5c59af799f39d3"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.304996 6913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:14:31.308317 6928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:31.309973 6913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:14:31.310097 6913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "b1a5a7a340224b9aa0caf9125e713120"
format_stamp: "Formatted at 2026-09-29 03:14:31 on dist-test-slave-k1dg"
server_key: "295996b12fc46de25fa0037409cd2c93"
server_key_iv: "8a065174df4d32e74c5c59af799f39d3"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.310285 6913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:31.326913 6913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:31.330858 6913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:31.331105 6913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:31.331789 6913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:31.332718 6913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:31.332779 6913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:31.332823 6913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:31.332882 6913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:31.342664 6913 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:36221
I20260929 03:14:31.342697 7041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:36221 every 8 connection(s)
I20260929 03:14:31.343674 6913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:14:31.351300 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 6913
I20260929 03:14:31.351464 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:14:31.351773 21177 external_mini_cluster.cc:1428] Setting key 0373bc9b05ee47c8758a295e23e706b9
Sep 29 03:14:31 dist-test-slave-k1dg krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:14:31.360618 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.345665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:50997 (local address 127.20.174.126:38107)
0929 03:14:31.345814 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:31.345818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:31.346824 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 03:14:31.348305 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:31.348309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:31.348363 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:31.348444 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:31.349820 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.350375 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.352289 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.352608 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.352747 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.355952 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:31.355956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:31.355961 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:31.356000 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:31.357828 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:31.358546 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:31.358549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:31.358550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:31.358716 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:31.359246 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:31.359249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:31.359250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:31.359459 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:31.359548 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:31.360349 (+ 801us) server_negotiation.cc:299] Negotiation successful
0929 03:14:31.360488 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:14:31.361917 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.346045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:38107 (local address 127.20.174.65:50997)
0929 03:14:31.346642 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:31.346693 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:31.348061 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:31.348639 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:31.348647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:31.349093 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:31.349634 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.349648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.350570 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.350573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.352069 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.352082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.352797 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.352800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.353008 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.353846 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:31.353864 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:31.355737 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:31.357952 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:31.357963 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:31.357987 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:31.358409 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:31.358839 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:31.358845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:31.358850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:31.359093 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:31.359586 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:31.359597 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:31.360027 (+ 430us) client_negotiation.cc:769] Sending connection context
0929 03:14:31.360269 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 03:14:31.360686 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:14:31.363646 7042 heartbeater.cc:347] Connected to a master server at 127.20.174.126:38107
I20260929 03:14:31.363998 7042 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:31.364553 7042 heartbeater.cc:511] Master 127.20.174.126:38107 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:14:31.366487 6851 ts_manager.cc:194] Registered new tserver with Master: b1a5a7a340224b9aa0caf9125e713120 (127.20.174.65:36221)
I20260929 03:14:31.367828 6851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:50997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:14:31.388170 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38107
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:31.506985 7047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:31.507265 7047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:31.507364 7047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:31.511632 7047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:31.511713 7047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:31.511777 7047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:14:31.517810 7047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:31.519068 7047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:31.520229 7047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:31.531369 7055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:31.531440 7047 server_base.cc:1034] running on GCE node
W20260929 03:14:31.531380 7052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:31.531394 7053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:31.532171 7047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:31.532864 7047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:31.534086 7047 hybrid_clock.cc:648] HybridClock initialized: now 1790651671534069 us; error 35 us; skew 500 ppm
Sep 29 03:14:31 dist-test-slave-k1dg krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:31.537410 7047 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:14:31.538591 7047 webserver.cc:460] Webserver started at http://127.20.174.66:43143/ using document root <none> and password file <none>
I20260929 03:14:31.539216 7047 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:31.539335 7047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:31.539597 7047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:31.542318 7047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "2ccdcfbce85c4104913d900f9c29a2a4"
format_stamp: "Formatted at 2026-09-29 03:14:31 on dist-test-slave-k1dg"
server_key: "76fca2a67d21e287259f76ce975441b7"
server_key_iv: "bf4dbf0354acdf02471ed705d752a3b9"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.543051 7047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "2ccdcfbce85c4104913d900f9c29a2a4"
format_stamp: "Formatted at 2026-09-29 03:14:31 on dist-test-slave-k1dg"
server_key: "76fca2a67d21e287259f76ce975441b7"
server_key_iv: "bf4dbf0354acdf02471ed705d752a3b9"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.547847 7047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:14:31.550747 7062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:31.552171 7047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:14:31.552345 7047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "2ccdcfbce85c4104913d900f9c29a2a4"
format_stamp: "Formatted at 2026-09-29 03:14:31 on dist-test-slave-k1dg"
server_key: "76fca2a67d21e287259f76ce975441b7"
server_key_iv: "bf4dbf0354acdf02471ed705d752a3b9"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.552493 7047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:31.564682 7047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:31.568455 7047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:31.568702 7047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:31.569415 7047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:31.570360 7047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:31.570438 7047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:31.570568 7047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:31.570621 7047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:31.580596 7047 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:43229
I20260929 03:14:31.580619 7175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:43229 every 8 connection(s)
I20260929 03:14:31.581661 7047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:14:31.585945 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 7047
I20260929 03:14:31.586084 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:14:31.586390 21177 external_mini_cluster.cc:1428] Setting key 5cd6888c570bc8ad0fb55ce4bd7e6b9d
Sep 29 03:14:31 dist-test-slave-k1dg krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:14:31.597978 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.583796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:51633 (local address 127.20.174.126:38107)
0929 03:14:31.583965 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:31.583969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:31.584864 (+ 895us) server_negotiation.cc:407] Connection header received
0929 03:14:31.586572 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:31.586579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:31.586641 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:31.586721 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:31.588187 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.588761 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.590366 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.590778 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.590905 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.593797 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:31.593804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:31.593809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:31.593848 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:31.595690 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:31.596289 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:31.596293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:31.596296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:31.596402 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:31.596750 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:31.596753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:31.596755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:31.597007 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:31.597120 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:31.597657 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 03:14:31.597773 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:14:31.599063 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.584121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:38107 (local address 127.20.174.66:51633)
0929 03:14:31.584687 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:31.584733 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:31.586268 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:31.586911 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:31.586918 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:31.587352 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:31.587981 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.588045 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.588912 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.588919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.590215 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.590226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.590953 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.590957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.591110 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.591760 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:31.591776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:31.593602 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:31.595829 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:31.595834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:31.595848 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:31.596173 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:31.596511 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:31.596513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:31.596514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:31.596653 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:31.597147 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:31.597152 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:31.597422 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 03:14:31.597590 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 03:14:31.598009 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":167,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:14:31.600541 7176 heartbeater.cc:347] Connected to a master server at 127.20.174.126:38107
I20260929 03:14:31.600840 7176 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:31.601511 7176 heartbeater.cc:511] Master 127.20.174.126:38107 requested a full tablet report, sending...
I20260929 03:14:31.602905 6851 ts_manager.cc:194] Registered new tserver with Master: 2ccdcfbce85c4104913d900f9c29a2a4 (127.20.174.66:43229)
I20260929 03:14:31.603715 6851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:51633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:14:31.618979 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38107
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:31.739391 7181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:31.739739 7181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:31.739830 7181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:31.744060 7181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:31.744179 7181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:31.744283 7181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:14:31.749854 7181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:38107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:31.751147 7181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:31.752241 7181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:31.762625 7186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:31.762689 7181 server_base.cc:1034] running on GCE node
W20260929 03:14:31.762640 7187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:31.762636 7189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:31.763437 7181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:31.764127 7181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:31.765345 7181 hybrid_clock.cc:648] HybridClock initialized: now 1790651671765304 us; error 56 us; skew 500 ppm
Sep 29 03:14:31 dist-test-slave-k1dg krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:31.768762 7181 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:14:31.769852 7181 webserver.cc:460] Webserver started at http://127.20.174.67:33457/ using document root <none> and password file <none>
I20260929 03:14:31.770509 7181 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:31.770583 7181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:31.770776 7181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:31.773102 7181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "0cafc2be2b0444a497bc01df7cf3e65b"
format_stamp: "Formatted at 2026-09-29 03:14:31 on dist-test-slave-k1dg"
server_key: "fc0f8255f2758ed0a1bd4abb308e8976"
server_key_iv: "9dc6a5a24231c7c2a5345405714dafd2"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.773744 7181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "0cafc2be2b0444a497bc01df7cf3e65b"
format_stamp: "Formatted at 2026-09-29 03:14:31 on dist-test-slave-k1dg"
server_key: "fc0f8255f2758ed0a1bd4abb308e8976"
server_key_iv: "9dc6a5a24231c7c2a5345405714dafd2"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.777891 7181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:14:31.780885 7196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:31.782541 7181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:31.782657 7181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "0cafc2be2b0444a497bc01df7cf3e65b"
format_stamp: "Formatted at 2026-09-29 03:14:31 on dist-test-slave-k1dg"
server_key: "fc0f8255f2758ed0a1bd4abb308e8976"
server_key_iv: "9dc6a5a24231c7c2a5345405714dafd2"
server_key_version: "encryptionkey@0"
I20260929 03:14:31.782850 7181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:31.812325 7181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:31.817129 7181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:31.817310 7181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:31.817921 7181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:31.818806 7181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:31.818857 7181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:31.818907 7181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:31.818924 7181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:31.828979 7181 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:34581
I20260929 03:14:31.829003 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:34581 every 8 connection(s)
I20260929 03:14:31.829952 7181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:14:31.836428 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 7181
I20260929 03:14:31.836588 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:14:31.836930 21177 external_mini_cluster.cc:1428] Setting key d625a87fd85fa4fa8b9760911aa4a35c
Sep 29 03:14:31 dist-test-slave-k1dg krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:14:31.845450 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.832156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:55017 (local address 127.20.174.126:38107)
0929 03:14:31.832336 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:31.832340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:31.833289 (+ 949us) server_negotiation.cc:407] Connection header received
0929 03:14:31.835003 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:31.835007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:31.835066 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:31.835190 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:31.836697 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.837382 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.838696 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.838925 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.839048 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.841736 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:31.841739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:31.841741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:31.841765 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:31.843216 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:31.843863 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:31.843866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:31.843867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:31.843949 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:31.844334 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:31.844338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:31.844340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:31.844537 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:31.844650 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:31.845160 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 03:14:31.845286 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 03:14:31.846318 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.832592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:38107 (local address 127.20.174.67:55017)
0929 03:14:31.833107 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:31.833158 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:31.834772 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:31.835369 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:31.835376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:31.835779 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:31.836409 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.836431 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.837539 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.837543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.838568 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.838579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.839089 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.839093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.839292 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.840004 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:31.840021 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:31.841552 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:31.843349 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:31.843356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:31.843372 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:31.843649 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:31.844062 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:31.844065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:31.844069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:31.844219 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:31.844647 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:31.844653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:31.844916 (+ 263us) client_negotiation.cc:769] Sending connection context
0929 03:14:31.845109 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 03:14:31.845457 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":123,"threads_started":1}
I20260929 03:14:31.847621 7310 heartbeater.cc:347] Connected to a master server at 127.20.174.126:38107
I20260929 03:14:31.848011 7310 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:31.848567 7310 heartbeater.cc:511] Master 127.20.174.126:38107 requested a full tablet report, sending...
I20260929 03:14:31.849707 6851 ts_manager.cc:194] Registered new tserver with Master: 0cafc2be2b0444a497bc01df7cf3e65b (127.20.174.67:34581)
I20260929 03:14:31.850329 6851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:55017
I20260929 03:14:31.852272 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:14:31.864534 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.854297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.20.174.126:38107)
0929 03:14:31.854406 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:31.854409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:31.854615 (+ 206us) server_negotiation.cc:407] Connection header received
0929 03:14:31.854719 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:31.854721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:31.854757 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:31.854830 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:31.855529 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.856133 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.857050 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.857291 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.857404 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.859128 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:31.859132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:31.859134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:31.859161 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:31.861099 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:31.862244 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:31.862248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:31.862249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:31.862542 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:31.863491 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:31.863494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:31.863496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:31.863814 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:31.863949 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:31.864316 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 03:14:31.864407 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:14:31.868391 6849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:14:31.871906 6849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:14:31.888595 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.882632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.20.174.66:43229)
0929 03:14:31.883034 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:31.883046 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:31.883222 (+ 176us) server_negotiation.cc:407] Connection header received
0929 03:14:31.883294 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:31.883298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:31.883447 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:31.883575 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:31.884041 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.885151 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.887034 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.887958 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.888088 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.888260 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:31.888318 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:14:31.888425 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":126,"threads_started":1}
I20260929 03:14:31.888669 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.881895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.20.174.67:34581)
0929 03:14:31.882578 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:31.882590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:31.882788 (+ 198us) server_negotiation.cc:407] Connection header received
0929 03:14:31.883103 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:31.883110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:31.883347 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:31.883479 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:31.883901 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.885082 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.886556 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.887497 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.887602 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.887870 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:31.888436 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 03:14:31.888536 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":86,"threads_started":1}
I20260929 03:14:31.888821 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.881775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:36221 (local address 127.0.0.1:57396)
0929 03:14:31.882721 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:31.882735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:31.882882 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:31.883533 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:31.883539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:31.883561 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:31.883711 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.883720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.885296 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.885300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.886888 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.886899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.888212 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.888215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.888314 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.888337 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:14:31.888387 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:14:31.888453 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":221,"spinlock_wait_cycles":9856,"thread_start_us":109,"threads_started":1}
I20260929 03:14:31.889060 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.881839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:43229 (local address 127.0.0.1:51708)
0929 03:14:31.883113 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:31.883125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:31.883210 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:31.883685 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:31.883691 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:31.883721 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:31.883855 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.883862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.885296 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.885300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.886888 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.886905 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.888086 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.888089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.888170 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.888199 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 03:14:31.888255 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:14:31.888428 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"mutex_wait_us":217,"thread_start_us":75,"threads_started":1}
I20260929 03:14:31.889060 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.881752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:34581 (local address 127.0.0.1:58036)
0929 03:14:31.882639 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:31.882696 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:31.882939 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:31.883655 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:31.883657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:31.883671 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:31.883743 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.883751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.885270 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.885273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.886341 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:31.886351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.887941 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.887944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:31.888074 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.888119 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 03:14:31.888235 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 03:14:31.888927 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":558,"spinlock_wait_cycles":1408,"thread_start_us":103,"threads_started":1}
I20260929 03:14:31.889247 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:31.882632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.20.174.65:36221)
0929 03:14:31.883034 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:31.883043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:31.883068 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:14:31.883146 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:31.883150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:31.883365 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:31.883524 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:31.883840 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.885151 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.887035 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:31.888074 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:31.888533 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:31.888682 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:31.888966 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 03:14:31.889076 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":123,"threads_started":1}
I20260929 03:14:31.891835 7109 tablet_service.cc:1467] Processing CreateTablet for tablet 15b89f3e493e4e21bd13181eab29dd69 (DEFAULT_TABLE table=test-table [id=2d74edc9e7074a9a9fd984f922f97743]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:31.891930 6976 tablet_service.cc:1467] Processing CreateTablet for tablet 15b89f3e493e4e21bd13181eab29dd69 (DEFAULT_TABLE table=test-table [id=2d74edc9e7074a9a9fd984f922f97743]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:31.891933 7244 tablet_service.cc:1467] Processing CreateTablet for tablet 15b89f3e493e4e21bd13181eab29dd69 (DEFAULT_TABLE table=test-table [id=2d74edc9e7074a9a9fd984f922f97743]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:31.892843 7109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15b89f3e493e4e21bd13181eab29dd69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:31.892931 7244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15b89f3e493e4e21bd13181eab29dd69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:31.892931 6976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15b89f3e493e4e21bd13181eab29dd69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:31.899519 7327 tablet_bootstrap.cc:492] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: Bootstrap starting.
I20260929 03:14:31.901113 7328 tablet_bootstrap.cc:492] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120: Bootstrap starting.
I20260929 03:14:31.902155 7327 tablet_bootstrap.cc:654] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:31.903507 7328 tablet_bootstrap.cc:654] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:31.903571 7327 log.cc:824] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:31.904431 7329 tablet_bootstrap.cc:492] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: Bootstrap starting.
I20260929 03:14:31.904402 7328 log.cc:824] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:31.906679 7328 tablet_bootstrap.cc:492] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120: No bootstrap required, opened a new log
I20260929 03:14:31.906749 7329 tablet_bootstrap.cc:654] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:31.906903 7328 ts_tablet_manager.cc:1397] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:14:31.908187 7327 tablet_bootstrap.cc:492] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: No bootstrap required, opened a new log
I20260929 03:14:31.908274 7329 log.cc:824] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:31.908454 7327 ts_tablet_manager.cc:1397] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 03:14:31.910413 7329 tablet_bootstrap.cc:492] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: No bootstrap required, opened a new log
I20260929 03:14:31.910629 7329 ts_tablet_manager.cc:1397] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260929 03:14:31.911667 7327 raft_consensus.cc:348] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43229 } } peers { permanent_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "b1a5a7a340224b9aa0caf9125e713120" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 36221 } }
I20260929 03:14:31.911677 7328 raft_consensus.cc:348] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43229 } } peers { permanent_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "b1a5a7a340224b9aa0caf9125e713120" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 36221 } }
I20260929 03:14:31.911927 7327 raft_consensus.cc:374] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:31.911947 7328 raft_consensus.cc:374] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:31.911983 7327 raft_consensus.cc:729] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccdcfbce85c4104913d900f9c29a2a4, State: Initialized, Role: FOLLOWER
I20260929 03:14:31.911994 7328 raft_consensus.cc:729] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a5a7a340224b9aa0caf9125e713120, State: Initialized, Role: FOLLOWER
I20260929 03:14:31.912457 7327 consensus_queue.cc:260] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43229 } } peers { permanent_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "b1a5a7a340224b9aa0caf9125e713120" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 36221 } }
I20260929 03:14:31.912462 7328 consensus_queue.cc:260] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43229 } } peers { permanent_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "b1a5a7a340224b9aa0caf9125e713120" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 36221 } }
I20260929 03:14:31.913493 7328 ts_tablet_manager.cc:1428] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 03:14:31.913509 7327 ts_tablet_manager.cc:1428] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260929 03:14:31.913944 7042 heartbeater.cc:503] Master 127.20.174.126:38107 was elected leader, sending a full tablet report...
I20260929 03:14:31.913923 7176 heartbeater.cc:503] Master 127.20.174.126:38107 was elected leader, sending a full tablet report...
I20260929 03:14:31.914638 7329 raft_consensus.cc:348] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43229 } } peers { permanent_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "b1a5a7a340224b9aa0caf9125e713120" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 36221 } }
I20260929 03:14:31.914953 7329 raft_consensus.cc:374] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:31.915009 7329 raft_consensus.cc:729] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cafc2be2b0444a497bc01df7cf3e65b, State: Initialized, Role: FOLLOWER
I20260929 03:14:31.915455 7329 consensus_queue.cc:260] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43229 } } peers { permanent_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "b1a5a7a340224b9aa0caf9125e713120" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 36221 } }
I20260929 03:14:31.916322 7329 ts_tablet_manager.cc:1428] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260929 03:14:31.916575 7310 heartbeater.cc:503] Master 127.20.174.126:38107 was elected leader, sending a full tablet report...
W20260929 03:14:32.081077 7311 tablet.cc:2369] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:14:32.082794 7177 tablet.cc:2369] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:14:32.095337 7043 tablet.cc:2369] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:14:32.102116 7333 raft_consensus.cc:482] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:14:32.102473 7333 raft_consensus.cc:504] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43229 } } peers { permanent_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "b1a5a7a340224b9aa0caf9125e713120" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 36221 } }
I20260929 03:14:32.103752 7333 leader_election.cc:288] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cafc2be2b0444a497bc01df7cf3e65b (127.20.174.67:34581), b1a5a7a340224b9aa0caf9125e713120 (127.20.174.65:36221)
I20260929 03:14:32.108290 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:32.104005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:34581 (local address 127.20.174.66:48119)
0929 03:14:32.104353 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:32.104368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:32.104484 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:32.104922 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:32.104925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:32.104942 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:32.105089 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:32.105097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.106050 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.106053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:32.107071 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:32.107080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.107804 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.107807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:32.107903 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:32.107929 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:14:32.108011 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:14:32.108078 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":28,"thread_start_us":103,"threads_started":1}
I20260929 03:14:32.108291 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:32.104116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:48119 (local address 127.20.174.67:34581)
0929 03:14:32.104313 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:32.104318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:32.104466 (+ 148us) server_negotiation.cc:407] Connection header received
0929 03:14:32.104591 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:32.104597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:32.104776 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:32.104840 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:32.105311 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.105916 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.107206 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.107683 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.107781 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:32.107822 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:32.108070 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 03:14:32.108154 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 03:14:32.108547 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:32.104224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:58687 (local address 127.20.174.65:36221)
0929 03:14:32.104347 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:32.104351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:32.104631 (+ 280us) server_negotiation.cc:407] Connection header received
0929 03:14:32.104890 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:32.104895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:32.104953 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:32.105048 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:32.105484 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.106108 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.107378 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.107921 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.108042 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:32.108086 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:32.108328 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 03:14:32.108414 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:14:32.108871 7337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:32.104115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:36221 (local address 127.20.174.66:58687)
0929 03:14:32.104500 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:32.104531 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:32.104648 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:32.105080 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:32.105082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:32.105097 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:32.105342 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:32.105350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.106248 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.106267 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:32.107249 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:32.107259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.108081 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.108084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:32.108183 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:32.108206 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:14:32.108291 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:14:32.108737 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":67,"thread_start_us":88,"threads_started":1}
I20260929 03:14:32.108942 7264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15b89f3e493e4e21bd13181eab29dd69" candidate_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" is_pre_election: true
I20260929 03:14:32.109266 7264 raft_consensus.cc:2454] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ccdcfbce85c4104913d900f9c29a2a4 in term 0.
I20260929 03:14:32.109522 6996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15b89f3e493e4e21bd13181eab29dd69" candidate_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a5a7a340224b9aa0caf9125e713120" is_pre_election: true
I20260929 03:14:32.109908 6996 raft_consensus.cc:2454] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ccdcfbce85c4104913d900f9c29a2a4 in term 0.
I20260929 03:14:32.109845 7065 leader_election.cc:302] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cafc2be2b0444a497bc01df7cf3e65b, 2ccdcfbce85c4104913d900f9c29a2a4; no voters:
I20260929 03:14:32.110190 7333 raft_consensus.cc:2781] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:14:32.110296 7333 raft_consensus.cc:482] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:14:32.110356 7333 raft_consensus.cc:3037] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:32.111465 7333 raft_consensus.cc:504] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43229 } } peers { permanent_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "b1a5a7a340224b9aa0caf9125e713120" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 36221 } }
I20260929 03:14:32.111858 7333 leader_election.cc:288] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [CANDIDATE]: Term 1 election: Requested vote from peers 0cafc2be2b0444a497bc01df7cf3e65b (127.20.174.67:34581), b1a5a7a340224b9aa0caf9125e713120 (127.20.174.65:36221)
I20260929 03:14:32.112259 7264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15b89f3e493e4e21bd13181eab29dd69" candidate_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cafc2be2b0444a497bc01df7cf3e65b"
I20260929 03:14:32.112277 6996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15b89f3e493e4e21bd13181eab29dd69" candidate_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a5a7a340224b9aa0caf9125e713120"
I20260929 03:14:32.112397 7264 raft_consensus.cc:3037] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:32.112409 6996 raft_consensus.cc:3037] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:32.113543 6996 raft_consensus.cc:2454] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ccdcfbce85c4104913d900f9c29a2a4 in term 1.
I20260929 03:14:32.113560 7264 raft_consensus.cc:2454] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ccdcfbce85c4104913d900f9c29a2a4 in term 1.
I20260929 03:14:32.113904 7065 leader_election.cc:302] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cafc2be2b0444a497bc01df7cf3e65b, 2ccdcfbce85c4104913d900f9c29a2a4; no voters:
I20260929 03:14:32.114073 7333 raft_consensus.cc:2781] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:32.114320 7333 raft_consensus.cc:686] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 2ccdcfbce85c4104913d900f9c29a2a4, State: Running, Role: LEADER
I20260929 03:14:32.114782 7333 consensus_queue.cc:237] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43229 } } peers { permanent_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } } peers { permanent_uuid: "b1a5a7a340224b9aa0caf9125e713120" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 36221 } }
I20260929 03:14:32.118113 6851 catalog_manager.cc:5495] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ccdcfbce85c4104913d900f9c29a2a4 (127.20.174.66). New cstate: current_term: 1 leader_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ccdcfbce85c4104913d900f9c29a2a4" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 43229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1a5a7a340224b9aa0caf9125e713120" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20260929 03:14:32.136668 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:32.132867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.20.174.66:43229)
0929 03:14:32.133028 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:32.133035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:32.133053 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:14:32.133126 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:32.133128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:32.133181 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:32.133297 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:14:32.133575 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.134122 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.135108 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.135334 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.135452 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:32.135781 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:14:32.136202 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:14:32.136372 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:32.136480 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 03:14:32.136520 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:14:32.142956 6996 raft_consensus.cc:1261] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120 [term 1 FOLLOWER]: Refusing update from remote peer 2ccdcfbce85c4104913d900f9c29a2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:32.143152 7264 raft_consensus.cc:1261] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b [term 1 FOLLOWER]: Refusing update from remote peer 2ccdcfbce85c4104913d900f9c29a2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:32.143831 7333 consensus_queue.cc:1035] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1a5a7a340224b9aa0caf9125e713120" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 36221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:32.144165 7338 consensus_queue.cc:1035] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cafc2be2b0444a497bc01df7cf3e65b" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 34581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:32.153717 7342 mvcc.cc:205] Tried to move back new op lower bound from 7334509249089687552 to 7334509248985157632. Current Snapshot: MvccSnapshot[applied={T|T < 7334509249089687552}]
I20260929 03:14:32.154623 7343 mvcc.cc:205] Tried to move back new op lower bound from 7334509249089687552 to 7334509248985157632. Current Snapshot: MvccSnapshot[applied={T|T < 7334509249089687552 or (T in {7334509249089687552})}]
I20260929 03:14:32.163538 6851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:32.163749 6851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:32.166538 6851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4ed6c22267b40228b6b6bb32726063d: Sending DeleteTablet for 3 replicas of tablet 15b89f3e493e4e21bd13181eab29dd69
I20260929 03:14:32.167478 7109 tablet_service.cc:1514] Processing DeleteTablet for tablet 15b89f3e493e4e21bd13181eab29dd69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:32 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:51708
I20260929 03:14:32.167479 7244 tablet_service.cc:1514] Processing DeleteTablet for tablet 15b89f3e493e4e21bd13181eab29dd69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:32 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:58036
I20260929 03:14:32.168110 6976 tablet_service.cc:1514] Processing DeleteTablet for tablet 15b89f3e493e4e21bd13181eab29dd69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:32 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:57396
I20260929 03:14:32.168340 7350 tablet_replica.cc:330] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: stopping tablet replica
I20260929 03:14:32.168618 7352 tablet_replica.cc:330] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120: stopping tablet replica
I20260929 03:14:32.168733 7350 raft_consensus.cc:2229] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:14:32.168794 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 6913
I20260929 03:14:32.168920 7352 raft_consensus.cc:2229] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:14:32.169185 7350 raft_consensus.cc:2258] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:32.169247 7352 raft_consensus.cc:2258] T 15b89f3e493e4e21bd13181eab29dd69 P b1a5a7a340224b9aa0caf9125e713120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:32.169647 7351 tablet_replica.cc:330] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: stopping tablet replica
I20260929 03:14:32.169978 7351 raft_consensus.cc:2229] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:14:32.170352 7351 raft_consensus.cc:2258] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:32.171052 7350 ts_tablet_manager.cc:1905] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:14:32.171972 7351 ts_tablet_manager.cc:1905] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:14:32.174088 7350 ts_tablet_manager.cc:1918] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:32.174194 7350 log.cc:1196] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/15b89f3e493e4e21bd13181eab29dd69
I20260929 03:14:32.174603 7350 ts_tablet_manager.cc:1939] T 15b89f3e493e4e21bd13181eab29dd69 P 2ccdcfbce85c4104913d900f9c29a2a4: Deleting consensus metadata
I20260929 03:14:32.175019 7351 ts_tablet_manager.cc:1918] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:32.175115 7351 log.cc:1196] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/15b89f3e493e4e21bd13181eab29dd69
I20260929 03:14:32.175451 7351 ts_tablet_manager.cc:1939] T 15b89f3e493e4e21bd13181eab29dd69 P 0cafc2be2b0444a497bc01df7cf3e65b: Deleting consensus metadata
I20260929 03:14:32.175962 6838 catalog_manager.cc:4841] TS 2ccdcfbce85c4104913d900f9c29a2a4 (127.20.174.66:43229): tablet 15b89f3e493e4e21bd13181eab29dd69 (table test-table [id=2d74edc9e7074a9a9fd984f922f97743]) successfully deleted
I20260929 03:14:32.176643 6837 catalog_manager.cc:4841] TS 0cafc2be2b0444a497bc01df7cf3e65b (127.20.174.67:34581): tablet 15b89f3e493e4e21bd13181eab29dd69 (table test-table [id=2d74edc9e7074a9a9fd984f922f97743]) successfully deleted
W20260929 03:14:32.180922 6836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:36221 (error 108)
I20260929 03:14:32.181865 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 7047
I20260929 03:14:32.182109 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:32.181762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:36221 (local address 127.0.0.1:57398)
0929 03:14:32.181944 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:32.182019 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:36221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 03:14:32.183145 6836 catalog_manager.cc:4584] TS b1a5a7a340224b9aa0caf9125e713120 (127.20.174.65:36221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15b89f3e493e4e21bd13181eab29dd69: Network error: Client connection negotiation failed: client connection to 127.20.174.65:36221: connect: Connection refused (error 111)
I20260929 03:14:32.188864 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 7181
I20260929 03:14:32.194684 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 6819
2026-09-29T03:14:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2560 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T03:14:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:32Z Disabled control of system clock
I20260929 03:14:32.226233 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:46743
--webserver_interface=127.20.174.126
--webserver_port=0
--builtin_ntp_servers=127.20.174.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:46743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:14:32.342703 7357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:32.342984 7357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:32.343021 7357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:32.347160 7357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:32.347235 7357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:32.347250 7357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:32.347268 7357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:32.347283 7357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:32.352294 7357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:46743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:46743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:32.353394 7357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:32.354792 7357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:32.363371 7362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:32.363493 7365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:32.363445 7357 server_base.cc:1034] running on GCE node
W20260929 03:14:32.363375 7363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:32.364133 7357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:32.365283 7357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:32.366515 7357 hybrid_clock.cc:648] HybridClock initialized: now 1790651672366479 us; error 53 us; skew 500 ppm
I20260929 03:14:32.368593 7357 webserver.cc:460] Webserver started at http://127.20.174.126:37733/ using document root <none> and password file <none>
I20260929 03:14:32.369210 7357 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:32.369308 7357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:32.369552 7357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:32.371944 7357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "770eaaa304054240be5749a14a3bb3b4"
format_stamp: "Formatted at 2026-09-29 03:14:32 on dist-test-slave-k1dg"
server_key: "77cc412ec386120af7e39df6566cd1ed"
server_key_iv: "9e7934d93b8b5dc67a537c0dde7578c4"
server_key_version: "encryptionkey@0"
I20260929 03:14:32.372568 7357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "770eaaa304054240be5749a14a3bb3b4"
format_stamp: "Formatted at 2026-09-29 03:14:32 on dist-test-slave-k1dg"
server_key: "77cc412ec386120af7e39df6566cd1ed"
server_key_iv: "9e7934d93b8b5dc67a537c0dde7578c4"
server_key_version: "encryptionkey@0"
I20260929 03:14:32.377628 7357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:14:32.380338 7371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:32.381656 7357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:14:32.381803 7357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "770eaaa304054240be5749a14a3bb3b4"
format_stamp: "Formatted at 2026-09-29 03:14:32 on dist-test-slave-k1dg"
server_key: "77cc412ec386120af7e39df6566cd1ed"
server_key_iv: "9e7934d93b8b5dc67a537c0dde7578c4"
server_key_version: "encryptionkey@0"
I20260929 03:14:32.381932 7357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:32.399662 7357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:32.400532 7357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:32.400741 7357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:32.407925 7357 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:46743
I20260929 03:14:32.407930 7423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:46743 every 8 connection(s)
I20260929 03:14:32.408963 7357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:32.412093 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:32.413236 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 7357
I20260929 03:14:32.413363 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:32.413704 21177 external_mini_cluster.cc:1428] Setting key 5de66b04e9ac3820ddc9b7dc7c46fbc7
I20260929 03:14:32.418653 7424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4: Bootstrap starting.
I20260929 03:14:32.420950 7424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:32.421700 7424 log.cc:824] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:32.423645 7427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:32.415273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.20.174.126:46743)
0929 03:14:32.416045 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:32.416062 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:32.416100 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:14:32.416763 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:32.416780 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:32.417215 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:32.417566 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:32.418016 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.419123 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.420069 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.420558 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.420735 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:32.421638 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:32.421643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:32.421657 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:32.422131 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:32.422203 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:32.422231 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:32.422336 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:32.422606 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 03:14:32.422908 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":181,"threads_started":1}
I20260929 03:14:32.424050 7424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4: No bootstrap required, opened a new log
I20260929 03:14:32.428169 7424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770eaaa304054240be5749a14a3bb3b4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46743 } }
I20260929 03:14:32.428458 7424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:32.428539 7424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770eaaa304054240be5749a14a3bb3b4, State: Initialized, Role: FOLLOWER
I20260929 03:14:32.429028 7424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770eaaa304054240be5749a14a3bb3b4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46743 } }
I20260929 03:14:32.429195 7424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:32.429283 7424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:32.429388 7424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:32.430631 7424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770eaaa304054240be5749a14a3bb3b4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46743 } }
I20260929 03:14:32.431001 7424 leader_election.cc:302] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 770eaaa304054240be5749a14a3bb3b4; no voters:
I20260929 03:14:32.431335 7424 leader_election.cc:288] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:32.431483 7429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:32.431731 7429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [term 1 LEADER]: Becoming Leader. State: Replica: 770eaaa304054240be5749a14a3bb3b4, State: Running, Role: LEADER
I20260929 03:14:32.432035 7429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770eaaa304054240be5749a14a3bb3b4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46743 } }
I20260929 03:14:32.432756 7424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:32.433912 7430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "770eaaa304054240be5749a14a3bb3b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770eaaa304054240be5749a14a3bb3b4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46743 } } }
I20260929 03:14:32.434051 7430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:32.434474 7438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:32.434422 7431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 770eaaa304054240be5749a14a3bb3b4. Latest consensus state: current_term: 1 leader_uuid: "770eaaa304054240be5749a14a3bb3b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770eaaa304054240be5749a14a3bb3b4" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46743 } } }
I20260929 03:14:32.434667 7431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:32.437667 7438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:32.443846 7438 catalog_manager.cc:1347] Generated new cluster ID: 80a12443a6834398934f82cb99379c55
I20260929 03:14:32.443960 7438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:32.453238 7438 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:32.454349 7438 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:32.462918 7438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4: Generated new TSK 0
I20260929 03:14:32.463688 7438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:32.468502 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:46743
--builtin_ntp_servers=127.20.174.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:14:32.589108 7448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:32.589460 7448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:32.589537 7448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:32.593942 7448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:32.594055 7448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:32.594156 7448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:14:32.599362 7448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:46743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:32.600589 7448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:32.602116 7448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:32.612871 7456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:32.612927 7454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:32.612937 7453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:32.613363 7448 server_base.cc:1034] running on GCE node
I20260929 03:14:32.613780 7448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:32.614579 7448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:32.615765 7448 hybrid_clock.cc:648] HybridClock initialized: now 1790651672615723 us; error 56 us; skew 500 ppm
I20260929 03:14:32.617771 7448 webserver.cc:460] Webserver started at http://127.20.174.65:39059/ using document root <none> and password file <none>
I20260929 03:14:32.618407 7448 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:32.618566 7448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:32.618763 7448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:32.620951 7448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "e9299c5a0852430fa7f2eed427cb7e4e"
format_stamp: "Formatted at 2026-09-29 03:14:32 on dist-test-slave-k1dg"
server_key: "3a91a1b16ffff2962f327450d768988c"
server_key_iv: "54336e8114f703942bf2b1f49116edae"
server_key_version: "encryptionkey@0"
I20260929 03:14:32.621497 7448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "e9299c5a0852430fa7f2eed427cb7e4e"
format_stamp: "Formatted at 2026-09-29 03:14:32 on dist-test-slave-k1dg"
server_key: "3a91a1b16ffff2962f327450d768988c"
server_key_iv: "54336e8114f703942bf2b1f49116edae"
server_key_version: "encryptionkey@0"
I20260929 03:14:32.625559 7448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 03:14:32.628329 7462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:32.629647 7448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:14:32.629750 7448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "e9299c5a0852430fa7f2eed427cb7e4e"
format_stamp: "Formatted at 2026-09-29 03:14:32 on dist-test-slave-k1dg"
server_key: "3a91a1b16ffff2962f327450d768988c"
server_key_iv: "54336e8114f703942bf2b1f49116edae"
server_key_version: "encryptionkey@0"
I20260929 03:14:32.629849 7448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:32.642321 7448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:32.643160 7448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:32.643319 7448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:32.643886 7448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:32.644891 7448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:32.644938 7448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:32.644980 7448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:32.644996 7448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:32.654138 7448 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:46649
I20260929 03:14:32.654227 7575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:46649 every 8 connection(s)
I20260929 03:14:32.655179 7448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:14:32.655848 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 7448
I20260929 03:14:32.656001 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:14:32.656352 21177 external_mini_cluster.cc:1428] Setting key 10bb8b9b45d5d8bc05185e7afd42b2a6
I20260929 03:14:32.658529 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:46743
--builtin_ntp_servers=127.20.174.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:14:32.666690 7427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:32.657501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:57385 (local address 127.20.174.126:46743)
0929 03:14:32.657649 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:32.657654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:32.659145 (+ 1491us) server_negotiation.cc:407] Connection header received
0929 03:14:32.660678 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:32.660683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:32.660749 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:32.660837 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:32.661907 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.662897 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.664146 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.664591 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.664736 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:32.666001 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:32.666008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:32.666012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:32.666032 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:32.666059 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:32.666072 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:32.666163 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:32.666333 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 03:14:32.666463 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:14:32.667711 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:32.658166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:46743 (local address 127.20.174.65:57385)
0929 03:14:32.658971 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:32.659030 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:32.660442 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:32.661025 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:32.661032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:32.661124 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:32.661724 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:32.661751 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.663072 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.663075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:32.664002 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:32.664013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.664790 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.664793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:32.664960 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:32.665664 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:14:32.665666 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:32.665710 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:14:32.665783 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:32.666175 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:32.666183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:32.666188 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:14:32.666299 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 03:14:32.666647 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":160,"threads_started":1}
I20260929 03:14:32.669178 7576 heartbeater.cc:347] Connected to a master server at 127.20.174.126:46743
I20260929 03:14:32.669461 7576 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:32.670034 7576 heartbeater.cc:511] Master 127.20.174.126:46743 requested a full tablet report, sending...
I20260929 03:14:32.671909 7388 ts_manager.cc:194] Registered new tserver with Master: e9299c5a0852430fa7f2eed427cb7e4e (127.20.174.65:46649)
I20260929 03:14:32.673599 7388 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.65:57385
W20260929 03:14:32.780799 7579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:32.781107 7579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:32.781147 7579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:32.785444 7579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:32.785557 7579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:32.785658 7579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:14:32.790974 7579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:46743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:32.792220 7579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:32.793680 7579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:32.804246 7585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:32.804227 7587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:32.804430 7579 server_base.cc:1034] running on GCE node
W20260929 03:14:32.804682 7584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:32.805219 7579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:32.805941 7579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:32.807127 7579 hybrid_clock.cc:648] HybridClock initialized: now 1790651672807108 us; error 37 us; skew 500 ppm
I20260929 03:14:32.809271 7579 webserver.cc:460] Webserver started at http://127.20.174.66:42045/ using document root <none> and password file <none>
I20260929 03:14:32.809901 7579 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:32.809993 7579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:32.810237 7579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:32.812444 7579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3"
format_stamp: "Formatted at 2026-09-29 03:14:32 on dist-test-slave-k1dg"
server_key: "45ac36d686700ebe0d914e0f086661a9"
server_key_iv: "a9be019f7277f879ad46b14d1918d915"
server_key_version: "encryptionkey@0"
I20260929 03:14:32.813036 7579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3"
format_stamp: "Formatted at 2026-09-29 03:14:32 on dist-test-slave-k1dg"
server_key: "45ac36d686700ebe0d914e0f086661a9"
server_key_iv: "a9be019f7277f879ad46b14d1918d915"
server_key_version: "encryptionkey@0"
I20260929 03:14:32.817142 7579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 03:14:32.820024 7593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:32.821544 7579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:14:32.821703 7579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3"
format_stamp: "Formatted at 2026-09-29 03:14:32 on dist-test-slave-k1dg"
server_key: "45ac36d686700ebe0d914e0f086661a9"
server_key_iv: "a9be019f7277f879ad46b14d1918d915"
server_key_version: "encryptionkey@0"
I20260929 03:14:32.821866 7579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:32.849462 7579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:32.850323 7579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:32.850587 7579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:32.851389 7579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:32.852447 7579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:32.852535 7579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:32.852612 7579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:32.852661 7579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:32.861799 7579 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:41981
I20260929 03:14:32.861830 7706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:41981 every 8 connection(s)
I20260929 03:14:32.862824 7579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:14:32.866297 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 7579
I20260929 03:14:32.866479 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:14:32.866788 21177 external_mini_cluster.cc:1428] Setting key 6f861cfcac5a249427bb6425224c4b83
I20260929 03:14:32.869403 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:46743
--builtin_ntp_servers=127.20.174.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:14:32.874734 7427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:32.864927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:49689 (local address 127.20.174.126:46743)
0929 03:14:32.865100 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:32.865104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:32.866519 (+ 1415us) server_negotiation.cc:407] Connection header received
0929 03:14:32.868454 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:32.868459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:32.868512 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:32.868617 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:32.870030 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.870628 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.872010 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.872271 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.872402 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:32.874054 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:32.874057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:32.874060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:32.874073 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:32.874095 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:32.874103 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:32.874244 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:32.874508 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 03:14:32.874576 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 03:14:32.875738 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:32.865362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:46743 (local address 127.20.174.66:49689)
0929 03:14:32.866275 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:32.866326 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:32.868145 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:32.868877 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:32.868888 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:32.869020 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:32.869858 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:32.869885 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.870781 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.870789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:32.871858 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:32.871868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:32.872452 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:32.872457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:32.872636 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:32.873540 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:14:32.873542 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:32.873587 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:14:32.873662 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:32.874246 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:32.874259 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:32.874272 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 03:14:32.874424 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 03:14:32.874858 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":124,"threads_started":1}
I20260929 03:14:32.877084 7707 heartbeater.cc:347] Connected to a master server at 127.20.174.126:46743
I20260929 03:14:32.877362 7707 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:32.877926 7707 heartbeater.cc:511] Master 127.20.174.126:46743 requested a full tablet report, sending...
I20260929 03:14:32.879221 7388 ts_manager.cc:194] Registered new tserver with Master: 8af8b0a3cf04413886f6f9c77b3d7fe3 (127.20.174.66:41981)
I20260929 03:14:32.880127 7388 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.66:49689
W20260929 03:14:32.989452 7710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:32.989764 7710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:32.989810 7710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:32.994846 7710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:32.994930 7710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:32.994992 7710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:14:33.000213 7710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:46743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:33.001355 7710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:33.002938 7710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:33.013469 7716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:33.013480 7715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:33.013480 7718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:33.013844 7710 server_base.cc:1034] running on GCE node
I20260929 03:14:33.014338 7710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:33.015094 7710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:33.016304 7710 hybrid_clock.cc:648] HybridClock initialized: now 1790651673016269 us; error 47 us; skew 500 ppm
I20260929 03:14:33.018929 7710 webserver.cc:460] Webserver started at http://127.20.174.67:42359/ using document root <none> and password file <none>
I20260929 03:14:33.019606 7710 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:33.019681 7710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:33.019922 7710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:33.022291 7710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "854b79daa215446bb568d6a0eaab2e30"
format_stamp: "Formatted at 2026-09-29 03:14:33 on dist-test-slave-k1dg"
server_key: "132e122b0a6df0cd124d8bdfa71c44be"
server_key_iv: "4c8c589a0920faa97ccf88aea130139b"
server_key_version: "encryptionkey@0"
I20260929 03:14:33.023021 7710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "854b79daa215446bb568d6a0eaab2e30"
format_stamp: "Formatted at 2026-09-29 03:14:33 on dist-test-slave-k1dg"
server_key: "132e122b0a6df0cd124d8bdfa71c44be"
server_key_iv: "4c8c589a0920faa97ccf88aea130139b"
server_key_version: "encryptionkey@0"
I20260929 03:14:33.027627 7710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:14:33.030900 7724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:33.032446 7710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:14:33.032568 7710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "854b79daa215446bb568d6a0eaab2e30"
format_stamp: "Formatted at 2026-09-29 03:14:33 on dist-test-slave-k1dg"
server_key: "132e122b0a6df0cd124d8bdfa71c44be"
server_key_iv: "4c8c589a0920faa97ccf88aea130139b"
server_key_version: "encryptionkey@0"
I20260929 03:14:33.032723 7710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:33.050100 7710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:33.050894 7710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:33.051088 7710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:33.051682 7710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:33.052623 7710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:33.052668 7710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:33.052706 7710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:33.052769 7710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:33.062552 7710 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:36057
I20260929 03:14:33.062595 7837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:36057 every 8 connection(s)
I20260929 03:14:33.063532 7710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:14:33.066625 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 7710
I20260929 03:14:33.066759 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:14:33.067057 21177 external_mini_cluster.cc:1428] Setting key 390438012047dae73867a1f58d366e94
I20260929 03:14:33.073777 7427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.065514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:60397 (local address 127.20.174.126:46743)
0929 03:14:33.065696 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:33.065701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:33.066976 (+ 1275us) server_negotiation.cc:407] Connection header received
0929 03:14:33.068833 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:33.068836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:33.068898 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:33.068971 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:33.069951 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.070520 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.071727 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.071994 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.072080 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.073271 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:33.073274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:33.073276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:33.073289 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:33.073309 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:33.073317 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:33.073394 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:33.073599 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 03:14:33.073652 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 03:14:33.074851 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.065963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:46743 (local address 127.20.174.67:60397)
0929 03:14:33.066800 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:33.066866 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:33.068603 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:33.069157 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:33.069165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:33.069271 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:33.069778 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.069791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.070642 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.070645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.071610 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.071620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.072127 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.072129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.072283 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.072994 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:14:33.072996 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:33.073039 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:14:33.073107 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:33.073432 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:33.073441 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:33.073446 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:14:33.073570 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 03:14:33.073803 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260929 03:14:33.076243 7838 heartbeater.cc:347] Connected to a master server at 127.20.174.126:46743
I20260929 03:14:33.076541 7838 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:33.077268 7838 heartbeater.cc:511] Master 127.20.174.126:46743 requested a full tablet report, sending...
I20260929 03:14:33.078549 7388 ts_manager.cc:194] Registered new tserver with Master: 854b79daa215446bb568d6a0eaab2e30 (127.20.174.67:36057)
I20260929 03:14:33.079156 7388 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.67:60397
I20260929 03:14:33.081195 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:14:33.088093 7427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.083265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.20.174.126:46743)
0929 03:14:33.083497 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:33.083501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:33.083675 (+ 174us) server_negotiation.cc:407] Connection header received
0929 03:14:33.083853 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:33.083862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:33.083911 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:33.084014 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:33.084392 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.084980 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.086059 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.086315 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.086501 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.087414 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:33.087418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:33.087421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:33.087435 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:33.087466 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:33.087475 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:33.087659 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:33.087928 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 03:14:33.087973 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260929 03:14:33.091343 7388 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:14:33.094760 7388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:14:33.110908 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.104826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:46649 (local address 127.0.0.1:52890)
0929 03:14:33.105889 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:33.105900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:33.106019 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:33.106554 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:33.106557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:33.106576 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:33.106652 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.106658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.107605 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.107610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.109152 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.109162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.110246 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.110250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.110373 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.110414 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 03:14:33.110566 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 03:14:33.110711 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":141,"thread_start_us":99,"threads_started":1}
I20260929 03:14:33.110996 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.105490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.20.174.65:46649)
0929 03:14:33.105813 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:33.105826 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:33.106000 (+ 174us) server_negotiation.cc:407] Connection header received
0929 03:14:33.106168 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:33.106175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:33.106393 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:33.106580 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:33.106743 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.107481 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.109308 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.110100 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.110217 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.110465 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:33.110615 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 03:14:33.110730 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
I20260929 03:14:33.111264 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.104302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.20.174.66:41981)
0929 03:14:33.105234 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:33.105246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:33.105569 (+ 323us) server_negotiation.cc:407] Connection header received
0929 03:14:33.105655 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:33.105663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:33.105861 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:33.106044 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:33.106307 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.107567 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.109669 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.110262 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.110367 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.110591 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:33.110979 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 03:14:33.111099 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":124,"threads_started":1}
I20260929 03:14:33.111552 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.104039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:36057 (local address 127.0.0.1:40038)
0929 03:14:33.104808 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:33.104865 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:33.105000 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:33.105942 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:33.105950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:33.105978 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:33.106134 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.106143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.107734 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.107738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.109427 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.109438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.110714 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.110717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.110820 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.110850 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:14:33.110958 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:14:33.111045 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":261,"thread_start_us":99,"threads_started":1}
I20260929 03:14:33.111979 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.104208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.20.174.67:36057)
0929 03:14:33.105206 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:33.105220 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:33.105250 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:14:33.105336 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:33.105341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:33.105580 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:33.105746 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:33.106287 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.107567 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.109559 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.110318 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.110416 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.110649 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:33.111207 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 03:14:33.111326 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":167,"threads_started":1}
I20260929 03:14:33.111569 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.104039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:41981 (local address 127.0.0.1:59122)
0929 03:14:33.105372 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:33.105385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:33.105476 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:33.106060 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:33.106066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:33.106084 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:33.106210 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.106217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.107694 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.107697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.109546 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.109553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.110649 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.110652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.110736 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.110756 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:14:33.110823 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:14:33.110880 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"mutex_wait_us":447,"thread_start_us":87,"threads_started":1}
I20260929 03:14:33.114796 7510 tablet_service.cc:1467] Processing CreateTablet for tablet 31b59c44d0d44a4a8e98a09aa7e7fe46 (DEFAULT_TABLE table=test-table [id=e5680a9b61aa4e3c99e11375297f3220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:33.114830 7772 tablet_service.cc:1467] Processing CreateTablet for tablet 31b59c44d0d44a4a8e98a09aa7e7fe46 (DEFAULT_TABLE table=test-table [id=e5680a9b61aa4e3c99e11375297f3220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:33.115247 7641 tablet_service.cc:1467] Processing CreateTablet for tablet 31b59c44d0d44a4a8e98a09aa7e7fe46 (DEFAULT_TABLE table=test-table [id=e5680a9b61aa4e3c99e11375297f3220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:33.115913 7510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b59c44d0d44a4a8e98a09aa7e7fe46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:33.115926 7772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b59c44d0d44a4a8e98a09aa7e7fe46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:33.116201 7641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b59c44d0d44a4a8e98a09aa7e7fe46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:33.122275 7855 tablet_bootstrap.cc:492] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e: Bootstrap starting.
I20260929 03:14:33.123895 7857 tablet_bootstrap.cc:492] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30: Bootstrap starting.
I20260929 03:14:33.125098 7855 tablet_bootstrap.cc:654] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:33.125818 7857 tablet_bootstrap.cc:654] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:33.125993 7855 log.cc:824] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:33.126647 7857 log.cc:824] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:33.126873 7856 tablet_bootstrap.cc:492] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3: Bootstrap starting.
I20260929 03:14:33.128103 7855 tablet_bootstrap.cc:492] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e: No bootstrap required, opened a new log
I20260929 03:14:33.128328 7855 ts_tablet_manager.cc:1397] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:14:33.128823 7856 tablet_bootstrap.cc:654] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:33.129640 7856 log.cc:824] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:33.131692 7856 tablet_bootstrap.cc:492] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3: No bootstrap required, opened a new log
I20260929 03:14:33.131847 7856 ts_tablet_manager.cc:1397] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 03:14:33.132191 7855 raft_consensus.cc:348] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b79daa215446bb568d6a0eaab2e30" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36057 } } peers { permanent_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41981 } } peers { permanent_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 46649 } }
I20260929 03:14:33.132454 7855 raft_consensus.cc:374] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:33.132537 7855 raft_consensus.cc:729] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9299c5a0852430fa7f2eed427cb7e4e, State: Initialized, Role: FOLLOWER
I20260929 03:14:33.133080 7855 consensus_queue.cc:260] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b79daa215446bb568d6a0eaab2e30" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36057 } } peers { permanent_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41981 } } peers { permanent_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 46649 } }
I20260929 03:14:33.133872 7576 heartbeater.cc:503] Master 127.20.174.126:46743 was elected leader, sending a full tablet report...
I20260929 03:14:33.134212 7855 ts_tablet_manager.cc:1428] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:14:33.136197 7857 tablet_bootstrap.cc:492] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30: No bootstrap required, opened a new log
I20260929 03:14:33.136042 7856 raft_consensus.cc:348] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b79daa215446bb568d6a0eaab2e30" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36057 } } peers { permanent_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41981 } } peers { permanent_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 46649 } }
I20260929 03:14:33.136314 7856 raft_consensus.cc:374] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:33.136389 7856 raft_consensus.cc:729] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af8b0a3cf04413886f6f9c77b3d7fe3, State: Initialized, Role: FOLLOWER
I20260929 03:14:33.136636 7857 ts_tablet_manager.cc:1397] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260929 03:14:33.137082 7856 consensus_queue.cc:260] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b79daa215446bb568d6a0eaab2e30" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36057 } } peers { permanent_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41981 } } peers { permanent_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 46649 } }
I20260929 03:14:33.138275 7856 ts_tablet_manager.cc:1428] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260929 03:14:33.138415 7707 heartbeater.cc:503] Master 127.20.174.126:46743 was elected leader, sending a full tablet report...
I20260929 03:14:33.141142 7857 raft_consensus.cc:348] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b79daa215446bb568d6a0eaab2e30" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36057 } } peers { permanent_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41981 } } peers { permanent_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 46649 } }
I20260929 03:14:33.141458 7857 raft_consensus.cc:374] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:33.141533 7857 raft_consensus.cc:729] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854b79daa215446bb568d6a0eaab2e30, State: Initialized, Role: FOLLOWER
I20260929 03:14:33.141969 7857 consensus_queue.cc:260] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b79daa215446bb568d6a0eaab2e30" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36057 } } peers { permanent_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41981 } } peers { permanent_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 46649 } }
I20260929 03:14:33.142760 7838 heartbeater.cc:503] Master 127.20.174.126:46743 was elected leader, sending a full tablet report...
I20260929 03:14:33.142932 7857 ts_tablet_manager.cc:1428] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260929 03:14:33.156469 7577 tablet.cc:2369] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:14:33.164045 7861 raft_consensus.cc:482] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:14:33.164285 7861 raft_consensus.cc:504] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b79daa215446bb568d6a0eaab2e30" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36057 } } peers { permanent_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41981 } } peers { permanent_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 46649 } }
I20260929 03:14:33.165479 7861 leader_election.cc:288] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 854b79daa215446bb568d6a0eaab2e30 (127.20.174.67:36057), 8af8b0a3cf04413886f6f9c77b3d7fe3 (127.20.174.66:41981)
I20260929 03:14:33.169466 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.165923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:34757 (local address 127.20.174.66:41981)
0929 03:14:33.166060 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:33.166064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:33.166080 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:14:33.166253 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:33.166256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:33.166309 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:33.166399 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:33.166703 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.167266 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.168375 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.168842 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.168973 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.169027 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:33.169254 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 03:14:33.169344 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:14:33.169461 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.165796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:41981 (local address 127.20.174.65:34757)
0929 03:14:33.165959 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:33.165977 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:33.166097 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:33.166414 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:33.166417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:33.166455 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:33.166584 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.166590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.167392 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.167397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.168230 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.168240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.168983 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.168986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.169099 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.169126 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 03:14:33.169211 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:14:33.169296 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 03:14:33.170122 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.165929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:35985 (local address 127.20.174.67:36057)
0929 03:14:33.166049 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:33.166054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:33.166502 (+ 448us) server_negotiation.cc:407] Connection header received
0929 03:14:33.166751 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:33.166753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:33.166798 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:33.166893 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:33.167161 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.167683 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.168758 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.169351 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.169458 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.169517 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:33.169940 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 03:14:33.170022 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:14:33.170250 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b59c44d0d44a4a8e98a09aa7e7fe46" candidate_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" is_pre_election: true
I20260929 03:14:33.170575 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.165796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:36057 (local address 127.20.174.65:35985)
0929 03:14:33.166380 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:33.166392 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:33.166562 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:33.166904 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:33.166906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:33.166922 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:33.167052 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.167057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.167799 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.167804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.168616 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:33.168625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.169620 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.169625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:33.169791 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.169817 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:14:33.170301 (+ 484us) client_negotiation.cc:240] Negotiation successful
0929 03:14:33.170377 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":208,"thread_start_us":164,"threads_started":1}
I20260929 03:14:33.170639 7661 raft_consensus.cc:2454] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9299c5a0852430fa7f2eed427cb7e4e in term 0.
I20260929 03:14:33.171160 7465 leader_election.cc:302] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8af8b0a3cf04413886f6f9c77b3d7fe3, e9299c5a0852430fa7f2eed427cb7e4e; no voters:
I20260929 03:14:33.171204 7792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b59c44d0d44a4a8e98a09aa7e7fe46" candidate_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854b79daa215446bb568d6a0eaab2e30" is_pre_election: true
I20260929 03:14:33.171468 7861 raft_consensus.cc:2781] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:14:33.171532 7792 raft_consensus.cc:2454] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9299c5a0852430fa7f2eed427cb7e4e in term 0.
I20260929 03:14:33.171577 7861 raft_consensus.cc:482] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:14:33.171624 7861 raft_consensus.cc:3037] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:33.172866 7861 raft_consensus.cc:504] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b79daa215446bb568d6a0eaab2e30" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36057 } } peers { permanent_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41981 } } peers { permanent_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 46649 } }
I20260929 03:14:33.173250 7861 leader_election.cc:288] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [CANDIDATE]: Term 1 election: Requested vote from peers 854b79daa215446bb568d6a0eaab2e30 (127.20.174.67:36057), 8af8b0a3cf04413886f6f9c77b3d7fe3 (127.20.174.66:41981)
I20260929 03:14:33.173614 7792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b59c44d0d44a4a8e98a09aa7e7fe46" candidate_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854b79daa215446bb568d6a0eaab2e30"
I20260929 03:14:33.173744 7792 raft_consensus.cc:3037] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:33.173797 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b59c44d0d44a4a8e98a09aa7e7fe46" candidate_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3"
I20260929 03:14:33.173967 7661 raft_consensus.cc:3037] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:33.174993 7792 raft_consensus.cc:2454] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9299c5a0852430fa7f2eed427cb7e4e in term 1.
I20260929 03:14:33.175282 7661 raft_consensus.cc:2454] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9299c5a0852430fa7f2eed427cb7e4e in term 1.
I20260929 03:14:33.175316 7466 leader_election.cc:302] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854b79daa215446bb568d6a0eaab2e30, e9299c5a0852430fa7f2eed427cb7e4e; no voters:
I20260929 03:14:33.175491 7861 raft_consensus.cc:2781] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:33.175737 7861 raft_consensus.cc:686] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [term 1 LEADER]: Becoming Leader. State: Replica: e9299c5a0852430fa7f2eed427cb7e4e, State: Running, Role: LEADER
I20260929 03:14:33.176093 7861 consensus_queue.cc:237] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b79daa215446bb568d6a0eaab2e30" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36057 } } peers { permanent_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41981 } } peers { permanent_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 46649 } }
I20260929 03:14:33.179662 7382 catalog_manager.cc:5495] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e reported cstate change: term changed from 0 to 1, leader changed from <none> to e9299c5a0852430fa7f2eed427cb7e4e (127.20.174.65). New cstate: current_term: 1 leader_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b79daa215446bb568d6a0eaab2e30" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9299c5a0852430fa7f2eed427cb7e4e" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 46649 } health_report { overall_health: HEALTHY } } }
I20260929 03:14:33.194844 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.190945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.20.174.65:46649)
0929 03:14:33.191098 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:33.191102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:33.191117 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:14:33.191315 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:33.191317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:33.191379 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:33.191503 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:14:33.191755 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.192253 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.193163 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:33.193389 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:33.193497 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:33.193860 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:14:33.194310 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:14:33.194505 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:33.194638 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 03:14:33.194678 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:14:33.201428 7661 raft_consensus.cc:1261] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3 [term 1 FOLLOWER]: Refusing update from remote peer e9299c5a0852430fa7f2eed427cb7e4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:33.201442 7792 raft_consensus.cc:1261] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30 [term 1 FOLLOWER]: Refusing update from remote peer e9299c5a0852430fa7f2eed427cb7e4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:33.202262 7865 consensus_queue.cc:1035] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8af8b0a3cf04413886f6f9c77b3d7fe3" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 41981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:33.202554 7861 consensus_queue.cc:1035] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P e9299c5a0852430fa7f2eed427cb7e4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "854b79daa215446bb568d6a0eaab2e30" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:33.213434 7869 mvcc.cc:205] Tried to move back new op lower bound from 7334509253424582656 to 7334509253332156416. Current Snapshot: MvccSnapshot[applied={T|T < 7334509253424582656 or (T in {7334509253424582656})}]
I20260929 03:14:33.220108 7388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:33.220273 7388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:33.222940 7388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 770eaaa304054240be5749a14a3bb3b4: Sending DeleteTablet for 3 replicas of tablet 31b59c44d0d44a4a8e98a09aa7e7fe46
I20260929 03:14:33.223794 7772 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b59c44d0d44a4a8e98a09aa7e7fe46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:33 UTC) from {username='slave'} at 127.0.0.1:40038
I20260929 03:14:33.223831 7641 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b59c44d0d44a4a8e98a09aa7e7fe46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:33 UTC) from {username='slave'} at 127.0.0.1:59122
I20260929 03:14:33.224153 7510 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b59c44d0d44a4a8e98a09aa7e7fe46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:33 UTC) from {username='slave'} at 127.0.0.1:52890
I20260929 03:14:33.224525 7878 tablet_replica.cc:330] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30: stopping tablet replica
I20260929 03:14:33.224876 7878 raft_consensus.cc:2229] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:14:33.224946 7879 tablet_replica.cc:330] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3: stopping tablet replica
I20260929 03:14:33.224993 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 7448
I20260929 03:14:33.225898 7878 raft_consensus.cc:2258] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:33.227175 7879 raft_consensus.cc:2229] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:14:33.227500 7879 raft_consensus.cc:2258] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:33.227675 7878 ts_tablet_manager.cc:1905] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:14:33.229064 7879 ts_tablet_manager.cc:1905] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:14:33.231279 7878 ts_tablet_manager.cc:1918] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:33.231396 7878 log.cc:1196] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/31b59c44d0d44a4a8e98a09aa7e7fe46
I20260929 03:14:33.231393 7879 ts_tablet_manager.cc:1918] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:33.231488 7879 log.cc:1196] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/31b59c44d0d44a4a8e98a09aa7e7fe46
I20260929 03:14:33.231730 7878 ts_tablet_manager.cc:1939] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 854b79daa215446bb568d6a0eaab2e30: Deleting consensus metadata
I20260929 03:14:33.231801 7879 ts_tablet_manager.cc:1939] T 31b59c44d0d44a4a8e98a09aa7e7fe46 P 8af8b0a3cf04413886f6f9c77b3d7fe3: Deleting consensus metadata
I20260929 03:14:33.233565 7375 catalog_manager.cc:4841] TS 854b79daa215446bb568d6a0eaab2e30 (127.20.174.67:36057): tablet 31b59c44d0d44a4a8e98a09aa7e7fe46 (table test-table [id=e5680a9b61aa4e3c99e11375297f3220]) successfully deleted
I20260929 03:14:33.233577 7374 catalog_manager.cc:4841] TS 8af8b0a3cf04413886f6f9c77b3d7fe3 (127.20.174.66:41981): tablet 31b59c44d0d44a4a8e98a09aa7e7fe46 (table test-table [id=e5680a9b61aa4e3c99e11375297f3220]) successfully deleted
W20260929 03:14:33.234453 7372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:46649 (error 108)
I20260929 03:14:33.234652 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 7579
I20260929 03:14:33.236304 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:33.235846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:46649 (local address 127.0.0.1:52906)
0929 03:14:33.236081 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:33.236187 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:46649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260929 03:14:33.236495 7372 catalog_manager.cc:4584] TS e9299c5a0852430fa7f2eed427cb7e4e (127.20.174.65:46649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31b59c44d0d44a4a8e98a09aa7e7fe46: Network error: Client connection negotiation failed: client connection to 127.20.174.65:46649: connect: Connection refused (error 111)
I20260929 03:14:33.240954 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 7710
I20260929 03:14:33.246251 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 7357
2026-09-29T03:14:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1050 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:33 dist-test-slave-k1dg krb5kdc[7884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:33 dist-test-slave-k1dg krb5kdc[7884](info): set up 2 sockets
Sep 29 03:14:33 dist-test-slave-k1dg krb5kdc[7884](info): commencing operation
krb5kdc: starting...
W20260929 03:14:34.318849 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:34 dist-test-slave-k1dg krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:14:34.451028 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:33265
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:33265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:34.567966 7897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:34.568369 7897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:34.568461 7897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:34.573391 7897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:34.573534 7897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:34.573580 7897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:34.573632 7897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:34.573681 7897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:34.580132 7897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:33265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:33265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:34.581449 7897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:34.582609 7897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:34.592464 7905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:34.592473 7903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:34.592492 7902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:34.592875 7897 server_base.cc:1034] running on GCE node
I20260929 03:14:34.593520 7897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:34.594895 7897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:34.596099 7897 hybrid_clock.cc:648] HybridClock initialized: now 1790651674596086 us; error 40 us; skew 500 ppm
Sep 29 03:14:34 dist-test-slave-k1dg krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651674, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:34.599313 7897 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:14:34.600327 7897 webserver.cc:460] Webserver started at http://127.20.174.126:39013/ using document root <none> and password file <none>
I20260929 03:14:34.600948 7897 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:34.601049 7897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:34.601289 7897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:34.603497 7897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "7dc91e85a7b847c3968197710578ed6e"
format_stamp: "Formatted at 2026-09-29 03:14:34 on dist-test-slave-k1dg"
server_key: "e7e46e920552f0b3884e4813e21499c5"
server_key_iv: "100c4020a0d9d57aaace33425994fa0d"
server_key_version: "encryptionkey@0"
I20260929 03:14:34.604250 7897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "7dc91e85a7b847c3968197710578ed6e"
format_stamp: "Formatted at 2026-09-29 03:14:34 on dist-test-slave-k1dg"
server_key: "e7e46e920552f0b3884e4813e21499c5"
server_key_iv: "100c4020a0d9d57aaace33425994fa0d"
server_key_version: "encryptionkey@0"
I20260929 03:14:34.608614 7897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:14:34.611341 7912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:34.612668 7897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:14:34.612833 7897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "7dc91e85a7b847c3968197710578ed6e"
format_stamp: "Formatted at 2026-09-29 03:14:34 on dist-test-slave-k1dg"
server_key: "e7e46e920552f0b3884e4813e21499c5"
server_key_iv: "100c4020a0d9d57aaace33425994fa0d"
server_key_version: "encryptionkey@0"
I20260929 03:14:34.612965 7897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:34.626653 7897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:34.630339 7897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:34.630615 7897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:34.638113 7897 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:33265
I20260929 03:14:34.638170 7964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:33265 every 8 connection(s)
I20260929 03:14:34.639219 7897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:34.642611 7965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:34.648285 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 7897
I20260929 03:14:34.648461 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:34.648804 21177 external_mini_cluster.cc:1428] Setting key cdce44b82f78da99a2646239c83eb3ef
I20260929 03:14:34.649196 7965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e: Bootstrap starting.
I20260929 03:14:34.652063 7965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:34.652918 7965 log.cc:824] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:34.655088 7965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e: No bootstrap required, opened a new log
Sep 29 03:14:34 dist-test-slave-k1dg krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:14:34.660032 7965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc91e85a7b847c3968197710578ed6e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33265 } }
I20260929 03:14:34.660286 7965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:34.660386 7965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc91e85a7b847c3968197710578ed6e, State: Initialized, Role: FOLLOWER
I20260929 03:14:34.660900 7965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc91e85a7b847c3968197710578ed6e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33265 } }
I20260929 03:14:34.661068 7965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:34.661155 7965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:34.661262 7965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:34.662283 7965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc91e85a7b847c3968197710578ed6e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33265 } }
I20260929 03:14:34.662719 7965 leader_election.cc:302] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dc91e85a7b847c3968197710578ed6e; no voters:
I20260929 03:14:34.663107 7965 leader_election.cc:288] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:34.663259 7970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:34.663534 7970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [term 1 LEADER]: Becoming Leader. State: Replica: 7dc91e85a7b847c3968197710578ed6e, State: Running, Role: LEADER
I20260929 03:14:34.663874 7970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc91e85a7b847c3968197710578ed6e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33265 } }
I20260929 03:14:34.664625 7965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:34.665272 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:34.650364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.20.174.126:33265)
0929 03:14:34.650887 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:34.650904 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:34.650942 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:14:34.651612 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:34.651631 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:34.651988 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:34.652400 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:34.653226 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:34.654203 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:34.655177 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:34.655673 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:34.655849 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:34.658920 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:34.658925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:34.658939 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:34.658966 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:34.660983 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:34.661653 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:34.661659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:34.661665 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:34.661810 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:34.662163 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:34.662167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:34.662169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:34.662760 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:34.662995 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:34.663839 (+ 844us) server_negotiation.cc:299] Negotiation successful
0929 03:14:34.664176 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":154,"threads_started":1}
I20260929 03:14:34.665546 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dc91e85a7b847c3968197710578ed6e. Latest consensus state: current_term: 1 leader_uuid: "7dc91e85a7b847c3968197710578ed6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc91e85a7b847c3968197710578ed6e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33265 } } }
I20260929 03:14:34.665670 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:34.665948 7971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dc91e85a7b847c3968197710578ed6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc91e85a7b847c3968197710578ed6e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 33265 } } }
I20260929 03:14:34.666047 7971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:34.666100 7979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:34.669243 7979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:34.675341 7979 catalog_manager.cc:1347] Generated new cluster ID: a392b3b3a1e643a18de4800fd2657e9f
I20260929 03:14:34.675434 7979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:34.689795 7979 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:34.691259 7979 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:34.701118 7979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e: Generated new TSK 0
I20260929 03:14:34.702014 7979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.174.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
Entry for principal kudu/127.20.174.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab.
I20260929 03:14:34.739563 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33265
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:34.855520 7991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:34.855824 7991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:34.855876 7991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:34.860098 7991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:34.860170 7991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:34.860239 7991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:14:34.866246 7991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.65.keytab
--principal=kudu/127.20.174.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:34.867519 7991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:34.868624 7991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:34.879379 7996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:34.879402 7997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:34.879866 7991 server_base.cc:1034] running on GCE node
W20260929 03:14:34.879992 7999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:34.880445 7991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:34.881166 7991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:34.882391 7991 hybrid_clock.cc:648] HybridClock initialized: now 1790651674882372 us; error 40 us; skew 500 ppm
Sep 29 03:14:34 dist-test-slave-k1dg krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651674, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:34.885672 7991 init.cc:377] Logged in from keytab as kudu/127.20.174.65@KRBTEST.COM (short username kudu)
I20260929 03:14:34.886757 7991 webserver.cc:460] Webserver started at http://127.20.174.65:38033/ using document root <none> and password file <none>
I20260929 03:14:34.887367 7991 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:34.887461 7991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:34.887719 7991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:34.889838 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "d16c30ac6c1544eabeb6377f4d7fe81c"
format_stamp: "Formatted at 2026-09-29 03:14:34 on dist-test-slave-k1dg"
server_key: "15245baa840b6f06cccf7c99b13304e6"
server_key_iv: "756ec407b14baef094351f911919d8ab"
server_key_version: "encryptionkey@0"
I20260929 03:14:34.890419 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "d16c30ac6c1544eabeb6377f4d7fe81c"
format_stamp: "Formatted at 2026-09-29 03:14:34 on dist-test-slave-k1dg"
server_key: "15245baa840b6f06cccf7c99b13304e6"
server_key_iv: "756ec407b14baef094351f911919d8ab"
server_key_version: "encryptionkey@0"
I20260929 03:14:34.894567 7991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 03:14:34.897341 8006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:34.898797 7991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 03:14:34.898943 7991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "d16c30ac6c1544eabeb6377f4d7fe81c"
format_stamp: "Formatted at 2026-09-29 03:14:34 on dist-test-slave-k1dg"
server_key: "15245baa840b6f06cccf7c99b13304e6"
server_key_iv: "756ec407b14baef094351f911919d8ab"
server_key_version: "encryptionkey@0"
I20260929 03:14:34.899068 7991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:34.933773 7991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:34.937369 7991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:34.937637 7991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:34.938294 7991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:34.939302 7991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:34.939391 7991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:34.939486 7991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:34.939540 7991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:34.949033 7991 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:41633
I20260929 03:14:34.949048 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:41633 every 8 connection(s)
I20260929 03:14:34.950018 7991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:14:34.957396 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 7991
I20260929 03:14:34.957607 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:14:34.957994 21177 external_mini_cluster.cc:1428] Setting key 3f0e7180ae21452ce6e556b39b192ecc
Sep 29 03:14:34 dist-test-slave-k1dg krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651674, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.65@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:14:34.965350 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:34.952052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:53853 (local address 127.20.174.126:33265)
0929 03:14:34.952191 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:34.952195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:34.953129 (+ 934us) server_negotiation.cc:407] Connection header received
0929 03:14:34.954581 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:34.954584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:34.954641 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:34.954757 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:34.956038 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:34.956581 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:34.957940 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:34.958236 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:34.958353 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:34.961346 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:34.961350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:34.961353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:34.961394 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:34.962995 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:34.963595 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:34.963599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:34.963601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:34.963693 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:34.964097 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:34.964100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:34.964102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:34.964316 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:34.964460 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:34.965039 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 03:14:34.965180 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:14:34.966356 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:34.952472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:33265 (local address 127.20.174.65:53853)
0929 03:14:34.952964 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:34.953010 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:34.954317 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:34.954960 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:34.954967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:34.955350 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:34.955858 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:34.955873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:34.956728 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:34.956732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:34.957788 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:34.957799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:34.958415 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:34.958419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:34.958638 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:34.959306 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:34.959324 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:34.961144 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:34.963142 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:34.963148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:34.963167 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:34.963469 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:34.963796 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:34.963800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:34.963802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:34.963974 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:34.964452 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:34.964459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:34.964749 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 03:14:34.964983 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 03:14:34.965348 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
I20260929 03:14:34.968012 8120 heartbeater.cc:347] Connected to a master server at 127.20.174.126:33265
WARNING: no policy specified for kudu/127.20.174.66@KRBTEST.COM; defaulting to no policy
I20260929 03:14:34.968286 8120 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:34.968842 8120 heartbeater.cc:511] Master 127.20.174.126:33265 requested a full tablet report, sending...
I20260929 03:14:34.970746 7929 ts_manager.cc:194] Registered new tserver with Master: d16c30ac6c1544eabeb6377f4d7fe81c (127.20.174.65:41633)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.66@KRBTEST.COM" created.
I20260929 03:14:34.972149 7929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.65@KRBTEST.COM'} at 127.20.174.65:53853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
Entry for principal kudu/127.20.174.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab.
I20260929 03:14:34.991441 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33265
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:35.110505 8125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:35.110833 8125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:35.110882 8125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:35.115329 8125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:35.115409 8125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:35.115474 8125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:14:35.121526 8125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.66.keytab
--principal=kudu/127.20.174.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:35.122861 8125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:35.123981 8125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:35.134518 8130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:35.134557 8133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:35.134557 8131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:35.134953 8125 server_base.cc:1034] running on GCE node
I20260929 03:14:35.135353 8125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:35.136003 8125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:35.137214 8125 hybrid_clock.cc:648] HybridClock initialized: now 1790651675137198 us; error 32 us; skew 500 ppm
Sep 29 03:14:35 dist-test-slave-k1dg krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651675, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:35.140403 8125 init.cc:377] Logged in from keytab as kudu/127.20.174.66@KRBTEST.COM (short username kudu)
I20260929 03:14:35.141578 8125 webserver.cc:460] Webserver started at http://127.20.174.66:39183/ using document root <none> and password file <none>
I20260929 03:14:35.142234 8125 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:35.142334 8125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:35.142621 8125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:35.144901 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "786b48823c1841469a6c872c7800faa0"
format_stamp: "Formatted at 2026-09-29 03:14:35 on dist-test-slave-k1dg"
server_key: "a66cf7f5345f0fdf24e1cde5b1d7fa15"
server_key_iv: "f1cb7598dc47336e8d6ea728850daf35"
server_key_version: "encryptionkey@0"
I20260929 03:14:35.145520 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "786b48823c1841469a6c872c7800faa0"
format_stamp: "Formatted at 2026-09-29 03:14:35 on dist-test-slave-k1dg"
server_key: "a66cf7f5345f0fdf24e1cde5b1d7fa15"
server_key_iv: "f1cb7598dc47336e8d6ea728850daf35"
server_key_version: "encryptionkey@0"
I20260929 03:14:35.149704 8125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 03:14:35.152493 8140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:35.153913 8125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:14:35.154074 8125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "786b48823c1841469a6c872c7800faa0"
format_stamp: "Formatted at 2026-09-29 03:14:35 on dist-test-slave-k1dg"
server_key: "a66cf7f5345f0fdf24e1cde5b1d7fa15"
server_key_iv: "f1cb7598dc47336e8d6ea728850daf35"
server_key_version: "encryptionkey@0"
I20260929 03:14:35.154204 8125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:35.166224 8125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:35.169976 8125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:35.170207 8125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:35.170958 8125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:35.171902 8125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:35.171977 8125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:35.172050 8125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:35.172101 8125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:35.182085 8125 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:44427
I20260929 03:14:35.182122 8253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:44427 every 8 connection(s)
I20260929 03:14:35.183128 8125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:14:35.188678 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 8125
I20260929 03:14:35.188813 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:14:35.189144 21177 external_mini_cluster.cc:1428] Setting key 8c46dddf1e7525f50ecbe7cf9bfdd03f
Sep 29 03:14:35 dist-test-slave-k1dg krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651675, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.66@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:14:35.199438 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.185057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:49641 (local address 127.20.174.126:33265)
0929 03:14:35.185225 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:35.185229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:35.186142 (+ 913us) server_negotiation.cc:407] Connection header received
0929 03:14:35.187666 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:35.187669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:35.187731 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:35.187816 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:35.189372 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.190013 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.191778 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.192028 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.192134 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.195297 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:35.195301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:35.195303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:35.195331 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:35.197047 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:35.197680 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:35.197683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:35.197685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:35.197777 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:35.198167 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:35.198171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:35.198173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:35.198377 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:35.198536 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:35.199115 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 03:14:35.199265 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.20.174.67@KRBTEST.COM; defaulting to no policy
I20260929 03:14:35.200526 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.185385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:33265 (local address 127.20.174.66:49641)
0929 03:14:35.185860 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:35.185917 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:35.187412 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:35.187999 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:35.188006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:35.188410 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:35.189190 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.189214 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.190159 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.190162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.191587 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.191606 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.192222 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.192230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.192505 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.193339 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:35.193358 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:35.195067 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:35.197192 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:35.197200 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:35.197218 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:35.197551 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:35.197889 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:35.197893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:35.197895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:35.198056 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:35.198534 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:35.198543 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:35.198834 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 03:14:35.199053 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 03:14:35.199445 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260929 03:14:35.202023 8254 heartbeater.cc:347] Connected to a master server at 127.20.174.126:33265
I20260929 03:14:35.202482 8254 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:35.203329 8254 heartbeater.cc:511] Master 127.20.174.126:33265 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.67@KRBTEST.COM" created.
I20260929 03:14:35.204846 7929 ts_manager.cc:194] Registered new tserver with Master: 786b48823c1841469a6c872c7800faa0 (127.20.174.66:44427)
I20260929 03:14:35.205765 7929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.66@KRBTEST.COM'} at 127.20.174.66:49641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
Entry for principal kudu/127.20.174.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab.
I20260929 03:14:35.228118 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33265
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:35.350975 8259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:35.351296 8259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:35.351356 8259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:35.355751 8259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:35.355827 8259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:35.355897 8259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:14:35.361840 8259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.67.keytab
--principal=kudu/127.20.174.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:33265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:35.363111 8259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:35.364209 8259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:14:35.375203 8259 server_base.cc:1034] running on GCE node
W20260929 03:14:35.375182 8265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:35.375155 8264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:35.375411 8267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:35.375928 8259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:35.376710 8259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:35.377920 8259 hybrid_clock.cc:648] HybridClock initialized: now 1790651675377882 us; error 54 us; skew 500 ppm
Sep 29 03:14:35 dist-test-slave-k1dg krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651675, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:35.381170 8259 init.cc:377] Logged in from keytab as kudu/127.20.174.67@KRBTEST.COM (short username kudu)
I20260929 03:14:35.382133 8259 webserver.cc:460] Webserver started at http://127.20.174.67:37081/ using document root <none> and password file <none>
I20260929 03:14:35.382844 8259 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:35.382937 8259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:35.383172 8259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:35.385342 8259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "40b08c906aae4f30bb952e810997952c"
format_stamp: "Formatted at 2026-09-29 03:14:35 on dist-test-slave-k1dg"
server_key: "a8e32c5563539b423d2dde36ba6d52b9"
server_key_iv: "8daa03f650744a1c3e70e815792d7d94"
server_key_version: "encryptionkey@0"
I20260929 03:14:35.386082 8259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "40b08c906aae4f30bb952e810997952c"
format_stamp: "Formatted at 2026-09-29 03:14:35 on dist-test-slave-k1dg"
server_key: "a8e32c5563539b423d2dde36ba6d52b9"
server_key_iv: "8daa03f650744a1c3e70e815792d7d94"
server_key_version: "encryptionkey@0"
I20260929 03:14:35.391304 8259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 03:14:35.394066 8274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:35.395725 8259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:35.395843 8259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "40b08c906aae4f30bb952e810997952c"
format_stamp: "Formatted at 2026-09-29 03:14:35 on dist-test-slave-k1dg"
server_key: "a8e32c5563539b423d2dde36ba6d52b9"
server_key_iv: "8daa03f650744a1c3e70e815792d7d94"
server_key_version: "encryptionkey@0"
I20260929 03:14:35.395951 8259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:35.440045 8259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:35.443823 8259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:35.444011 8259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:35.444648 8259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:35.445511 8259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:35.445559 8259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:35.445600 8259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:35.445616 8259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:35.456079 8259 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:41403
I20260929 03:14:35.456121 8387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:41403 every 8 connection(s)
I20260929 03:14:35.457103 8259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:14:35.466524 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 8259
I20260929 03:14:35.466683 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:14:35.467029 21177 external_mini_cluster.cc:1428] Setting key 82c9067f4979b1681707f41c90477893
Sep 29 03:14:35 dist-test-slave-k1dg krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651675, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.67@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:14:35.473448 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.459290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:48675 (local address 127.20.174.126:33265)
0929 03:14:35.459478 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:35.459483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:35.460500 (+ 1017us) server_negotiation.cc:407] Connection header received
0929 03:14:35.461983 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:35.461986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:35.462037 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:35.462121 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:35.463492 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.464055 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.465510 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.465741 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.465833 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.469143 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:35.469146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:35.469148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:35.469175 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:35.471189 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:35.471788 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:35.471791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:35.471792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:35.471874 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:35.472260 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:35.472263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:35.472264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:35.472512 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:35.472607 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:35.473171 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 03:14:35.473290 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:14:35.474336 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.459784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:33265 (local address 127.20.174.67:48675)
0929 03:14:35.460325 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:35.460373 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:35.461642 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:35.462276 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:35.462284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:35.462786 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:35.463319 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.463335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.464211 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.464215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.465391 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.465403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.465902 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.465904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.466056 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.466929 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:14:35.466950 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:35.468945 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:35.471342 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:35.471348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:35.471363 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:35.471655 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:35.471967 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:14:35.471969 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:14:35.471971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:14:35.472133 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:14:35.472629 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:35.472634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:35.472927 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 03:14:35.473099 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 03:14:35.473424 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":128,"threads_started":1}
I20260929 03:14:35.475836 8388 heartbeater.cc:347] Connected to a master server at 127.20.174.126:33265
I20260929 03:14:35.476198 8388 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:35.476771 8388 heartbeater.cc:511] Master 127.20.174.126:33265 requested a full tablet report, sending...
I20260929 03:14:35.477937 7929 ts_manager.cc:194] Registered new tserver with Master: 40b08c906aae4f30bb952e810997952c (127.20.174.67:41403)
I20260929 03:14:35.478586 7929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.174.67@KRBTEST.COM'} at 127.20.174.67:48675
I20260929 03:14:35.481506 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:14:35.492173 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.483607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.20.174.126:33265)
0929 03:14:35.483788 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:35.483793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:35.484029 (+ 236us) server_negotiation.cc:407] Connection header received
0929 03:14:35.484214 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:35.484223 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:35.484277 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:35.484375 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:35.485189 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.485766 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.486819 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.487063 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.487188 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.488764 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:35.488775 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:35.488778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:35.488803 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:35.490137 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:35.490716 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:35.490720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:35.490722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:35.490815 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:35.491133 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:35.491138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:35.491140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:35.491415 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:35.491579 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:35.491852 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 03:14:35.491985 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:14:35.495494 7929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:14:35.498731 7929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:14:35.514838 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.508710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:41633 (local address 127.0.0.1:41424)
0929 03:14:35.509268 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:35.509283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:35.509408 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:35.510193 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:35.510199 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:35.510222 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:35.510365 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.510372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.511629 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.511633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.513030 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.513041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.514213 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.514217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.514334 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.514378 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 03:14:35.514520 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 03:14:35.514676 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260929 03:14:35.515205 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.508342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:44427 (local address 127.0.0.1:40110)
0929 03:14:35.509015 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:35.509069 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:35.509711 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:35.510363 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:35.510366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:35.510391 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:35.510546 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.510552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.512090 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.512094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.513467 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.513478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.514765 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.514769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.514865 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.514891 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:14:35.514966 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:14:35.515057 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":48,"spinlock_wait_cycles":768,"thread_start_us":112,"threads_started":1}
I20260929 03:14:35.515405 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.508547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.20.174.67:41403)
0929 03:14:35.509274 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:35.509288 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:35.509548 (+ 260us) server_negotiation.cc:407] Connection header received
0929 03:14:35.509790 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:35.509795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:35.509996 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:35.510139 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:35.510790 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.511926 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.513747 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.514590 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.514698 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.514828 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:35.515162 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 03:14:35.515270 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":141,"threads_started":1}
I20260929 03:14:35.515424 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.509229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.20.174.65:41633)
0929 03:14:35.509637 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:35.509644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:35.509667 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:14:35.509754 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:35.509760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:35.509996 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:35.510151 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:35.510539 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.511481 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.513174 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.514056 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.514181 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.514352 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:35.515200 (+ 848us) server_negotiation.cc:299] Negotiation successful
0929 03:14:35.515293 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":92,"threads_started":1}
I20260929 03:14:35.515782 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.508876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:41403 (local address 127.0.0.1:47858)
0929 03:14:35.509441 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:35.509457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:35.509633 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:35.510343 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:35.510345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:35.510362 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:35.510649 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.510657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.512090 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.512094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.513619 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.513633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.514890 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.514897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.514994 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.515016 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:14:35.515562 (+ 546us) client_negotiation.cc:240] Negotiation successful
0929 03:14:35.515653 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":99,"spinlock_wait_cycles":370560,"thread_start_us":95,"threads_started":1}
I20260929 03:14:35.516317 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.508374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.20.174.66:44427)
0929 03:14:35.508783 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:35.508795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:35.509564 (+ 769us) server_negotiation.cc:407] Connection header received
0929 03:14:35.509874 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:35.509879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:35.510031 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:35.510210 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:35.510703 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.511946 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.513616 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.514626 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.514766 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.515221 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:35.516012 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 03:14:35.516141 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":94,"threads_started":1}
I20260929 03:14:35.518841 8054 tablet_service.cc:1467] Processing CreateTablet for tablet 296cc22377a04bd1876e4be1b77bbf6a (DEFAULT_TABLE table=test-table [id=c185fa22b9c04fceb95abbee1f137ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:35.518846 8322 tablet_service.cc:1467] Processing CreateTablet for tablet 296cc22377a04bd1876e4be1b77bbf6a (DEFAULT_TABLE table=test-table [id=c185fa22b9c04fceb95abbee1f137ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:35.519054 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 296cc22377a04bd1876e4be1b77bbf6a (DEFAULT_TABLE table=test-table [id=c185fa22b9c04fceb95abbee1f137ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:35.519907 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296cc22377a04bd1876e4be1b77bbf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:35.519920 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296cc22377a04bd1876e4be1b77bbf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:35.519928 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296cc22377a04bd1876e4be1b77bbf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:35.527112 8405 tablet_bootstrap.cc:492] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c: Bootstrap starting.
I20260929 03:14:35.528458 8407 tablet_bootstrap.cc:492] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: Bootstrap starting.
I20260929 03:14:35.530169 8405 tablet_bootstrap.cc:654] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:35.530354 8407 tablet_bootstrap.cc:654] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:35.531136 8405 log.cc:824] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:35.531491 8407 log.cc:824] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:35.531970 8406 tablet_bootstrap.cc:492] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: Bootstrap starting.
I20260929 03:14:35.533192 8405 tablet_bootstrap.cc:492] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c: No bootstrap required, opened a new log
I20260929 03:14:35.533412 8405 ts_tablet_manager.cc:1397] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:14:35.533632 8407 tablet_bootstrap.cc:492] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: No bootstrap required, opened a new log
I20260929 03:14:35.533850 8407 ts_tablet_manager.cc:1397] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:14:35.534385 8406 tablet_bootstrap.cc:654] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:35.535753 8406 log.cc:824] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:35.537582 8405 raft_consensus.cc:348] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786b48823c1841469a6c872c7800faa0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44427 } } peers { permanent_uuid: "40b08c906aae4f30bb952e810997952c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41403 } } peers { permanent_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 41633 } }
I20260929 03:14:35.537849 8405 raft_consensus.cc:374] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:35.537904 8405 raft_consensus.cc:729] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d16c30ac6c1544eabeb6377f4d7fe81c, State: Initialized, Role: FOLLOWER
I20260929 03:14:35.537995 8406 tablet_bootstrap.cc:492] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: No bootstrap required, opened a new log
I20260929 03:14:35.537855 8407 raft_consensus.cc:348] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786b48823c1841469a6c872c7800faa0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44427 } } peers { permanent_uuid: "40b08c906aae4f30bb952e810997952c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41403 } } peers { permanent_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 41633 } }
I20260929 03:14:35.538133 8407 raft_consensus.cc:374] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:35.538199 8406 ts_tablet_manager.cc:1397] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:14:35.538198 8407 raft_consensus.cc:729] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786b48823c1841469a6c872c7800faa0, State: Initialized, Role: FOLLOWER
I20260929 03:14:35.538559 8405 consensus_queue.cc:260] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786b48823c1841469a6c872c7800faa0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44427 } } peers { permanent_uuid: "40b08c906aae4f30bb952e810997952c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41403 } } peers { permanent_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 41633 } }
I20260929 03:14:35.538669 8407 consensus_queue.cc:260] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786b48823c1841469a6c872c7800faa0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44427 } } peers { permanent_uuid: "40b08c906aae4f30bb952e810997952c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41403 } } peers { permanent_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 41633 } }
I20260929 03:14:35.539215 8120 heartbeater.cc:503] Master 127.20.174.126:33265 was elected leader, sending a full tablet report...
I20260929 03:14:35.539592 8405 ts_tablet_manager.cc:1428] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:14:35.539602 8407 ts_tablet_manager.cc:1428] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:14:35.540329 8254 heartbeater.cc:503] Master 127.20.174.126:33265 was elected leader, sending a full tablet report...
I20260929 03:14:35.542004 8406 raft_consensus.cc:348] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786b48823c1841469a6c872c7800faa0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44427 } } peers { permanent_uuid: "40b08c906aae4f30bb952e810997952c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41403 } } peers { permanent_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 41633 } }
I20260929 03:14:35.542347 8406 raft_consensus.cc:374] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:35.542411 8406 raft_consensus.cc:729] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b08c906aae4f30bb952e810997952c, State: Initialized, Role: FOLLOWER
I20260929 03:14:35.542986 8406 consensus_queue.cc:260] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786b48823c1841469a6c872c7800faa0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44427 } } peers { permanent_uuid: "40b08c906aae4f30bb952e810997952c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41403 } } peers { permanent_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 41633 } }
I20260929 03:14:35.543751 8388 heartbeater.cc:503] Master 127.20.174.126:33265 was elected leader, sending a full tablet report...
I20260929 03:14:35.544013 8406 ts_tablet_manager.cc:1428] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:14:35.660521 8413 raft_consensus.cc:482] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:14:35.660856 8413 raft_consensus.cc:504] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786b48823c1841469a6c872c7800faa0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44427 } } peers { permanent_uuid: "40b08c906aae4f30bb952e810997952c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41403 } } peers { permanent_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 41633 } }
I20260929 03:14:35.662154 8413 leader_election.cc:288] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 786b48823c1841469a6c872c7800faa0 (127.20.174.66:44427), d16c30ac6c1544eabeb6377f4d7fe81c (127.20.174.65:41633)
I20260929 03:14:35.667554 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.662535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:46187 (local address 127.20.174.66:44427)
0929 03:14:35.662745 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:35.662753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:35.662773 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:14:35.662837 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:35.662840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:35.662908 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:35.662983 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:35.663329 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.664212 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.666042 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.666723 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.666891 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.666947 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:35.667339 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 03:14:35.667415 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:14:35.667554 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.662714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:51973 (local address 127.20.174.65:41633)
0929 03:14:35.662840 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:35.662846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:35.663045 (+ 199us) server_negotiation.cc:407] Connection header received
0929 03:14:35.663121 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:35.663125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:35.663179 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:35.663246 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:35.663520 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.664363 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.666040 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.666706 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.666909 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.666951 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:35.667339 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 03:14:35.667420 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:14:35.667609 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.662370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:44427 (local address 127.20.174.67:46187)
0929 03:14:35.662529 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:35.662546 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:35.662652 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:35.663029 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:35.663032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:35.663045 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:35.663184 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.663189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.664346 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.664349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.665923 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.665929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.666901 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.666905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.667170 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.667219 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 03:14:35.667303 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:14:35.667453 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":222}
I20260929 03:14:35.667618 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.662527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:41633 (local address 127.20.174.67:51973)
0929 03:14:35.662913 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:35.662925 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:35.663042 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:35.663289 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:35.663294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:35.663314 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:35.663409 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.663414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.664564 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.664567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.665901 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:35.665917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.666919 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.666923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:35.667165 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.667217 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 03:14:35.667303 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:14:35.667449 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":183,"thread_start_us":183,"threads_started":1}
I20260929 03:14:35.668430 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296cc22377a04bd1876e4be1b77bbf6a" candidate_uuid: "40b08c906aae4f30bb952e810997952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786b48823c1841469a6c872c7800faa0" is_pre_election: true
I20260929 03:14:35.668807 8208 raft_consensus.cc:2454] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40b08c906aae4f30bb952e810997952c in term 0.
I20260929 03:14:35.669370 8275 leader_election.cc:302] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40b08c906aae4f30bb952e810997952c, 786b48823c1841469a6c872c7800faa0; no voters:
I20260929 03:14:35.669549 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296cc22377a04bd1876e4be1b77bbf6a" candidate_uuid: "40b08c906aae4f30bb952e810997952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" is_pre_election: true
I20260929 03:14:35.669677 8413 raft_consensus.cc:2781] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:14:35.669772 8413 raft_consensus.cc:482] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:14:35.669828 8413 raft_consensus.cc:3037] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:35.669888 8074 raft_consensus.cc:2454] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40b08c906aae4f30bb952e810997952c in term 0.
I20260929 03:14:35.671136 8413 raft_consensus.cc:504] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786b48823c1841469a6c872c7800faa0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44427 } } peers { permanent_uuid: "40b08c906aae4f30bb952e810997952c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41403 } } peers { permanent_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 41633 } }
I20260929 03:14:35.671568 8413 leader_election.cc:288] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [CANDIDATE]: Term 1 election: Requested vote from peers 786b48823c1841469a6c872c7800faa0 (127.20.174.66:44427), d16c30ac6c1544eabeb6377f4d7fe81c (127.20.174.65:41633)
I20260929 03:14:35.672030 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296cc22377a04bd1876e4be1b77bbf6a" candidate_uuid: "40b08c906aae4f30bb952e810997952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c"
I20260929 03:14:35.672030 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296cc22377a04bd1876e4be1b77bbf6a" candidate_uuid: "40b08c906aae4f30bb952e810997952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786b48823c1841469a6c872c7800faa0"
I20260929 03:14:35.672178 8208 raft_consensus.cc:3037] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:35.672169 8074 raft_consensus.cc:3037] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:35.673514 8208 raft_consensus.cc:2454] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40b08c906aae4f30bb952e810997952c in term 1.
I20260929 03:14:35.673514 8074 raft_consensus.cc:2454] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40b08c906aae4f30bb952e810997952c in term 1.
I20260929 03:14:35.673933 8277 leader_election.cc:302] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40b08c906aae4f30bb952e810997952c, d16c30ac6c1544eabeb6377f4d7fe81c; no voters:
I20260929 03:14:35.674119 8413 raft_consensus.cc:2781] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:35.674337 8413 raft_consensus.cc:686] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 1 LEADER]: Becoming Leader. State: Replica: 40b08c906aae4f30bb952e810997952c, State: Running, Role: LEADER
I20260929 03:14:35.674763 8413 consensus_queue.cc:237] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786b48823c1841469a6c872c7800faa0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44427 } } peers { permanent_uuid: "40b08c906aae4f30bb952e810997952c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41403 } } peers { permanent_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 41633 } }
I20260929 03:14:35.678349 7929 catalog_manager.cc:5495] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c reported cstate change: term changed from 0 to 1, leader changed from <none> to 40b08c906aae4f30bb952e810997952c (127.20.174.67). New cstate: current_term: 1 leader_uuid: "40b08c906aae4f30bb952e810997952c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786b48823c1841469a6c872c7800faa0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40b08c906aae4f30bb952e810997952c" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 41403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 41633 } health_report { overall_health: UNKNOWN } } }
W20260929 03:14:35.684506 8255 tablet.cc:2369] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:14:35.701514 8121 tablet.cc:2369] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:14:35.708244 8389 tablet.cc:2369] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:14:35.715123 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.711179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.20.174.67:41403)
0929 03:14:35.711351 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:35.711356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:35.711372 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:14:35.711526 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:35.711529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:35.711586 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:35.711710 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:14:35.712055 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.712589 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.713515 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:35.713737 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:35.713856 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:35.714188 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:14:35.714661 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:14:35.714808 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:35.714886 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 03:14:35.714950 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 03:14:35.721745 8074 raft_consensus.cc:1261] T 296cc22377a04bd1876e4be1b77bbf6a P d16c30ac6c1544eabeb6377f4d7fe81c [term 1 FOLLOWER]: Refusing update from remote peer 40b08c906aae4f30bb952e810997952c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:35.721832 8208 raft_consensus.cc:1261] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0 [term 1 FOLLOWER]: Refusing update from remote peer 40b08c906aae4f30bb952e810997952c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:35.722621 8413 consensus_queue.cc:1035] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d16c30ac6c1544eabeb6377f4d7fe81c" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 41633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:35.722898 8415 consensus_queue.cc:1035] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [LEADER]: Connected to new peer: Peer: permanent_uuid: "786b48823c1841469a6c872c7800faa0" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:35.741696 7929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:35.741971 7929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:35.744982 7929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dc91e85a7b847c3968197710578ed6e: Sending DeleteTablet for 3 replicas of tablet 296cc22377a04bd1876e4be1b77bbf6a
I20260929 03:14:35.745851 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 296cc22377a04bd1876e4be1b77bbf6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:35 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:40110
I20260929 03:14:35.745923 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet 296cc22377a04bd1876e4be1b77bbf6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:35 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:41424
I20260929 03:14:35.745942 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet 296cc22377a04bd1876e4be1b77bbf6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:35 UTC) from {username='kudu', principal='kudu/127.20.174.126@KRBTEST.COM'} at 127.0.0.1:47858
I20260929 03:14:35.746456 8428 tablet_replica.cc:330] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: stopping tablet replica
I20260929 03:14:35.746769 8430 tablet_replica.cc:330] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: stopping tablet replica
I20260929 03:14:35.746814 8428 raft_consensus.cc:2229] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:14:35.746897 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 7991
I20260929 03:14:35.747166 8428 raft_consensus.cc:2258] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:35.747323 8430 raft_consensus.cc:2229] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:14:35.747822 8430 raft_consensus.cc:2258] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:35.749043 8428 ts_tablet_manager.cc:1905] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:14:35.749576 8430 ts_tablet_manager.cc:1905] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:14:35.752003 8428 ts_tablet_manager.cc:1918] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:35.752121 8428 log.cc:1196] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/296cc22377a04bd1876e4be1b77bbf6a
I20260929 03:14:35.752120 8430 ts_tablet_manager.cc:1918] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:35.752220 8430 log.cc:1196] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/296cc22377a04bd1876e4be1b77bbf6a
I20260929 03:14:35.752463 8428 ts_tablet_manager.cc:1939] T 296cc22377a04bd1876e4be1b77bbf6a P 786b48823c1841469a6c872c7800faa0: Deleting consensus metadata
I20260929 03:14:35.752583 8430 ts_tablet_manager.cc:1939] T 296cc22377a04bd1876e4be1b77bbf6a P 40b08c906aae4f30bb952e810997952c: Deleting consensus metadata
I20260929 03:14:35.753849 7916 catalog_manager.cc:4841] TS 40b08c906aae4f30bb952e810997952c (127.20.174.67:41403): tablet 296cc22377a04bd1876e4be1b77bbf6a (table test-table [id=c185fa22b9c04fceb95abbee1f137ef6]) successfully deleted
I20260929 03:14:35.753980 7913 catalog_manager.cc:4841] TS 786b48823c1841469a6c872c7800faa0 (127.20.174.66:44427): tablet 296cc22377a04bd1876e4be1b77bbf6a (table test-table [id=c185fa22b9c04fceb95abbee1f137ef6]) successfully deleted
I20260929 03:14:35.759754 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 8125
W20260929 03:14:35.759706 7915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:41633 (error 108)
I20260929 03:14:35.761806 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:35.761391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:41633 (local address 127.0.0.1:41430)
0929 03:14:35.761563 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:35.761660 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:41633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:14:35.762115 7915 catalog_manager.cc:4584] TS d16c30ac6c1544eabeb6377f4d7fe81c (127.20.174.65:41633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 296cc22377a04bd1876e4be1b77bbf6a: Network error: Client connection negotiation failed: client connection to 127.20.174.65:41633: connect: Connection refused (error 111)
I20260929 03:14:35.766307 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 8259
I20260929 03:14:35.772243 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 7897
2026-09-29T03:14:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2527 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T03:14:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:35Z Disabled control of system clock
I20260929 03:14:35.804059 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40341
--webserver_interface=127.20.174.126
--webserver_port=0
--builtin_ntp_servers=127.20.174.84:39385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:40341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:14:35.923223 8435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:35.923539 8435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:35.923578 8435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:35.927978 8435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:35.928061 8435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:35.928076 8435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:35.928094 8435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:35.928110 8435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:35.933139 8435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:40341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:35.934252 8435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:35.935771 8435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:35.945343 8441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:35.945315 8440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:35.945595 8435 server_base.cc:1034] running on GCE node
W20260929 03:14:35.945312 8443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:35.946280 8435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:35.947613 8435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:35.948875 8435 hybrid_clock.cc:648] HybridClock initialized: now 1790651675948849 us; error 37 us; skew 500 ppm
I20260929 03:14:35.951313 8435 webserver.cc:460] Webserver started at http://127.20.174.126:45873/ using document root <none> and password file <none>
I20260929 03:14:35.952033 8435 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:35.952144 8435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:35.952400 8435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:35.954732 8435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "978d6d0364a2447bb10e9d3e762c80ca"
format_stamp: "Formatted at 2026-09-29 03:14:35 on dist-test-slave-k1dg"
server_key: "f15113e4001faecf3709f5a136a0babc"
server_key_iv: "5c7ddb77ce64686f59d7714f376ce2f4"
server_key_version: "encryptionkey@0"
I20260929 03:14:35.955340 8435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "978d6d0364a2447bb10e9d3e762c80ca"
format_stamp: "Formatted at 2026-09-29 03:14:35 on dist-test-slave-k1dg"
server_key: "f15113e4001faecf3709f5a136a0babc"
server_key_iv: "5c7ddb77ce64686f59d7714f376ce2f4"
server_key_version: "encryptionkey@0"
I20260929 03:14:35.959599 8435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:14:35.962634 8449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:35.964300 8435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:14:35.964481 8435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "978d6d0364a2447bb10e9d3e762c80ca"
format_stamp: "Formatted at 2026-09-29 03:14:35 on dist-test-slave-k1dg"
server_key: "f15113e4001faecf3709f5a136a0babc"
server_key_iv: "5c7ddb77ce64686f59d7714f376ce2f4"
server_key_version: "encryptionkey@0"
I20260929 03:14:35.964623 8435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:35.982844 8435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:35.983628 8435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:35.983850 8435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:35.991631 8501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:40341 every 8 connection(s)
I20260929 03:14:35.991626 8435 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:40341
I20260929 03:14:35.992708 8435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:35.996145 8502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:36.001050 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 8435
I20260929 03:14:36.001227 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:36.001566 21177 external_mini_cluster.cc:1428] Setting key db7b39ce2a3584e51d23df8b1c8a9096
I20260929 03:14:36.003253 8502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca: Bootstrap starting.
I20260929 03:14:36.005800 8502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:36.006711 8502 log.cc:824] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:36.009186 8502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca: No bootstrap required, opened a new log
I20260929 03:14:36.013249 8502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978d6d0364a2447bb10e9d3e762c80ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40341 } }
I20260929 03:14:36.013516 8502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:36.013550 8502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978d6d0364a2447bb10e9d3e762c80ca, State: Initialized, Role: FOLLOWER
I20260929 03:14:36.013947 8502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978d6d0364a2447bb10e9d3e762c80ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40341 } }
I20260929 03:14:36.014058 8502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:36.014098 8502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:36.014163 8502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:36.015240 8502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978d6d0364a2447bb10e9d3e762c80ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40341 } }
I20260929 03:14:36.015575 8502 leader_election.cc:302] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 978d6d0364a2447bb10e9d3e762c80ca; no voters:
I20260929 03:14:36.015911 8502 leader_election.cc:288] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:36.016009 8507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:36.016224 8507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [term 1 LEADER]: Becoming Leader. State: Replica: 978d6d0364a2447bb10e9d3e762c80ca, State: Running, Role: LEADER
I20260929 03:14:36.016546 8507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978d6d0364a2447bb10e9d3e762c80ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40341 } }
I20260929 03:14:36.017392 8502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:36.017854 8509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 978d6d0364a2447bb10e9d3e762c80ca. Latest consensus state: current_term: 1 leader_uuid: "978d6d0364a2447bb10e9d3e762c80ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978d6d0364a2447bb10e9d3e762c80ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40341 } } }
I20260929 03:14:36.017985 8509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:36.018575 8512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:36.019413 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.003230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.20.174.126:40341)
0929 03:14:36.004046 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:36.004064 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:36.004105 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:14:36.004803 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:36.004822 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:36.005276 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:36.005592 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:36.006062 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.007268 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.008582 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.009089 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.009217 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.016965 (+ 7748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:36.016972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:36.016991 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:36.017443 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:36.017549 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:36.017586 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:36.017704 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:36.017902 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 03:14:36.018218 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":169,"threads_started":1}
I20260929 03:14:36.020251 8508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "978d6d0364a2447bb10e9d3e762c80ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978d6d0364a2447bb10e9d3e762c80ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40341 } } }
I20260929 03:14:36.020358 8508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:36.022329 8512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:36.028175 8512 catalog_manager.cc:1347] Generated new cluster ID: be3aaa4ccc864f758f0b92b57c3083fe
I20260929 03:14:36.028270 8512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:36.046802 8512 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:36.048233 8512 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:36.061985 8512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca: Generated new TSK 0
I20260929 03:14:36.062892 8512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:36.068879 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.65:0
--local_ip_for_outbound_sockets=127.20.174.65
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40341
--builtin_ntp_servers=127.20.174.84:39385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:14:36.197664 8526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:36.197983 8526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:36.198050 8526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:36.202418 8526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:36.202541 8526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:36.202603 8526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.65
I20260929 03:14:36.207890 8526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.174.65
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.65
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:36.209151 8526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:36.210808 8526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:36.222218 8534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:36.222210 8531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:36.222337 8532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:36.222472 8526 server_base.cc:1034] running on GCE node
I20260929 03:14:36.223153 8526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:36.223942 8526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:36.225121 8526 hybrid_clock.cc:648] HybridClock initialized: now 1790651676225101 us; error 37 us; skew 500 ppm
I20260929 03:14:36.227443 8526 webserver.cc:460] Webserver started at http://127.20.174.65:33423/ using document root <none> and password file <none>
I20260929 03:14:36.228155 8526 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:36.228228 8526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:36.228470 8526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:36.230887 8526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/data/instance:
uuid: "ef4f66ce9269455fa680560500beffba"
format_stamp: "Formatted at 2026-09-29 03:14:36 on dist-test-slave-k1dg"
server_key: "e3f1669e271c7a770f26e225006e94b9"
server_key_iv: "4e33e34b9416a0c6db09f7e20d7fa1af"
server_key_version: "encryptionkey@0"
I20260929 03:14:36.231527 8526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance:
uuid: "ef4f66ce9269455fa680560500beffba"
format_stamp: "Formatted at 2026-09-29 03:14:36 on dist-test-slave-k1dg"
server_key: "e3f1669e271c7a770f26e225006e94b9"
server_key_iv: "4e33e34b9416a0c6db09f7e20d7fa1af"
server_key_version: "encryptionkey@0"
I20260929 03:14:36.235848 8526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:14:36.238940 8540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:36.240461 8526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:14:36.240619 8526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/wal
uuid: "ef4f66ce9269455fa680560500beffba"
format_stamp: "Formatted at 2026-09-29 03:14:36 on dist-test-slave-k1dg"
server_key: "e3f1669e271c7a770f26e225006e94b9"
server_key_iv: "4e33e34b9416a0c6db09f7e20d7fa1af"
server_key_version: "encryptionkey@0"
I20260929 03:14:36.240757 8526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:36.290012 8526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:36.290884 8526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:36.291115 8526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:36.291810 8526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:36.292889 8526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:36.292977 8526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:36.293049 8526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:36.293100 8526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:36.302995 8526 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.65:43543
I20260929 03:14:36.303028 8653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.65:43543 every 8 connection(s)
I20260929 03:14:36.304010 8526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/data/info.pb
I20260929 03:14:36.307370 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 8526
I20260929 03:14:36.307515 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-0/wal/instance
I20260929 03:14:36.307958 21177 external_mini_cluster.cc:1428] Setting key c9db4cb40d36505d250cc80f2a44be93
I20260929 03:14:36.310617 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.66:0
--local_ip_for_outbound_sockets=127.20.174.66
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40341
--builtin_ntp_servers=127.20.174.84:39385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:14:36.316510 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.306153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.65:58357 (local address 127.20.174.126:40341)
0929 03:14:36.306305 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:36.306309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:36.307594 (+ 1285us) server_negotiation.cc:407] Connection header received
0929 03:14:36.309539 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:36.309542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:36.309608 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:36.309678 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:36.311051 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.311987 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.313876 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.314426 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.314607 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.315814 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:36.315819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:36.315823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:36.315848 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:36.315880 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:36.315897 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:36.315993 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:36.316161 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:14:36.316261 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:14:36.317390 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.306662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40341 (local address 127.20.174.65:58357)
0929 03:14:36.307395 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:36.307474 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:36.309230 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:36.309930 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:36.309939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:36.310040 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:36.310883 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.310901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.312166 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.312172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.313699 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.313711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.314628 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.314631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.314794 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.315504 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:14:36.315505 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:36.315553 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:14:36.315633 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:36.316007 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:36.316014 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:36.316019 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:14:36.316125 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:14:36.316390 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260929 03:14:36.319099 8654 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40341
I20260929 03:14:36.319438 8654 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:36.320035 8654 heartbeater.cc:511] Master 127.20.174.126:40341 requested a full tablet report, sending...
I20260929 03:14:36.321847 8466 ts_manager.cc:194] Registered new tserver with Master: ef4f66ce9269455fa680560500beffba (127.20.174.65:43543)
I20260929 03:14:36.323146 8466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.65:58357
W20260929 03:14:36.436578 8657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:36.436940 8657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:36.437012 8657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:36.441493 8657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:36.441610 8657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:36.441720 8657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.66
I20260929 03:14:36.447278 8657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.174.66
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.66
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:36.448612 8657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:36.450275 8657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:36.461814 8662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:36.461764 8657 server_base.cc:1034] running on GCE node
W20260929 03:14:36.461745 8665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:36.461808 8663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:36.462617 8657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:36.463392 8657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:36.464578 8657 hybrid_clock.cc:648] HybridClock initialized: now 1790651676464508 us; error 85 us; skew 500 ppm
I20260929 03:14:36.466861 8657 webserver.cc:460] Webserver started at http://127.20.174.66:44903/ using document root <none> and password file <none>
I20260929 03:14:36.467523 8657 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:36.467595 8657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:36.467834 8657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:36.470322 8657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/data/instance:
uuid: "8f59d7c35be2434aaf9b9544f7aaec45"
format_stamp: "Formatted at 2026-09-29 03:14:36 on dist-test-slave-k1dg"
server_key: "3a00846da908e7e3f5d741a61c753c43"
server_key_iv: "2305380372adf00608752405411eec65"
server_key_version: "encryptionkey@0"
I20260929 03:14:36.471019 8657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance:
uuid: "8f59d7c35be2434aaf9b9544f7aaec45"
format_stamp: "Formatted at 2026-09-29 03:14:36 on dist-test-slave-k1dg"
server_key: "3a00846da908e7e3f5d741a61c753c43"
server_key_iv: "2305380372adf00608752405411eec65"
server_key_version: "encryptionkey@0"
I20260929 03:14:36.475291 8657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:14:36.478310 8671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:36.480047 8657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:36.480170 8657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/wal
uuid: "8f59d7c35be2434aaf9b9544f7aaec45"
format_stamp: "Formatted at 2026-09-29 03:14:36 on dist-test-slave-k1dg"
server_key: "3a00846da908e7e3f5d741a61c753c43"
server_key_iv: "2305380372adf00608752405411eec65"
server_key_version: "encryptionkey@0"
I20260929 03:14:36.480355 8657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:36.513931 8657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:36.514810 8657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:36.515040 8657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:36.515918 8657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:36.517117 8657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:36.517171 8657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:36.517244 8657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:36.517287 8657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:36.528025 8657 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.66:46125
I20260929 03:14:36.528043 8784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.66:46125 every 8 connection(s)
I20260929 03:14:36.529076 8657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/data/info.pb
I20260929 03:14:36.529342 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 8657
I20260929 03:14:36.529480 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/wal/instance
I20260929 03:14:36.529939 21177 external_mini_cluster.cc:1428] Setting key 102aae478322cdc9dffd6b8c365f1669
I20260929 03:14:36.532411 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.174.67:0
--local_ip_for_outbound_sockets=127.20.174.67
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40341
--builtin_ntp_servers=127.20.174.84:39385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:14:36.542084 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.531466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:42795 (local address 127.20.174.126:40341)
0929 03:14:36.531618 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:36.531623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:36.533205 (+ 1582us) server_negotiation.cc:407] Connection header received
0929 03:14:36.535108 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:36.535112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:36.535173 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:36.535258 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:36.536580 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.537232 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.539368 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.539596 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.539696 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.541337 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:36.541341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:36.541343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:36.541360 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:36.541393 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:36.541402 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:36.541473 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:36.541699 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 03:14:36.541822 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:14:36.543859 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.532101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40341 (local address 127.20.174.66:42795)
0929 03:14:36.533006 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:36.533078 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:36.534863 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:36.535462 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:36.535473 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:36.535597 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:36.536373 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.536397 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.537414 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.537421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.539198 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.539215 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.539810 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.539816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.540095 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.540914 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:14:36.540918 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:36.540994 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:14:36.541112 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:36.541509 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:36.541522 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:36.541530 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:14:36.541698 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 03:14:36.542309 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":141,"threads_started":1}
I20260929 03:14:36.545344 8785 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40341
I20260929 03:14:36.545627 8785 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:36.546307 8785 heartbeater.cc:511] Master 127.20.174.126:40341 requested a full tablet report, sending...
I20260929 03:14:36.547566 8466 ts_manager.cc:194] Registered new tserver with Master: 8f59d7c35be2434aaf9b9544f7aaec45 (127.20.174.66:46125)
I20260929 03:14:36.548221 8466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.66:42795
W20260929 03:14:36.670912 8788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:36.671197 8788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:36.671240 8788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:36.675486 8788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:36.675585 8788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:36.675669 8788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.174.67
I20260929 03:14:36.681198 8788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.174.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.174.67
--webserver_port=0
--tserver_master_addrs=127.20.174.126:40341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.174.67
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:36.682422 8788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:36.683986 8788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:36.694519 8793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:36.694514 8796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:36.694600 8788 server_base.cc:1034] running on GCE node
W20260929 03:14:36.694514 8794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:36.695374 8788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:36.696115 8788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:36.697357 8788 hybrid_clock.cc:648] HybridClock initialized: now 1790651676697342 us; error 36 us; skew 500 ppm
I20260929 03:14:36.699663 8788 webserver.cc:460] Webserver started at http://127.20.174.67:38589/ using document root <none> and password file <none>
I20260929 03:14:36.700294 8788 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:36.700393 8788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:36.700626 8788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:36.702884 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/data/instance:
uuid: "f185c62a0c4a47e0851f6a8fa16d744f"
format_stamp: "Formatted at 2026-09-29 03:14:36 on dist-test-slave-k1dg"
server_key: "061452e016b288ead10d28813109e0a3"
server_key_iv: "45cf31b865e91b822f80559e947ff8bc"
server_key_version: "encryptionkey@0"
I20260929 03:14:36.703563 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance:
uuid: "f185c62a0c4a47e0851f6a8fa16d744f"
format_stamp: "Formatted at 2026-09-29 03:14:36 on dist-test-slave-k1dg"
server_key: "061452e016b288ead10d28813109e0a3"
server_key_iv: "45cf31b865e91b822f80559e947ff8bc"
server_key_version: "encryptionkey@0"
I20260929 03:14:36.707597 8788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:14:36.710227 8802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:36.711624 8788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:36.711766 8788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/wal
uuid: "f185c62a0c4a47e0851f6a8fa16d744f"
format_stamp: "Formatted at 2026-09-29 03:14:36 on dist-test-slave-k1dg"
server_key: "061452e016b288ead10d28813109e0a3"
server_key_iv: "45cf31b865e91b822f80559e947ff8bc"
server_key_version: "encryptionkey@0"
I20260929 03:14:36.711891 8788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:36.748584 8788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:36.749377 8788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:36.749601 8788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:36.750254 8788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:14:36.751411 8788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:14:36.751497 8788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:36.751571 8788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:14:36.751606 8788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:36.761262 8788 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.67:39791
I20260929 03:14:36.761302 8915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.67:39791 every 8 connection(s)
I20260929 03:14:36.762248 8788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/data/info.pb
I20260929 03:14:36.771298 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 8788
I20260929 03:14:36.771443 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/wal/instance
I20260929 03:14:36.771821 21177 external_mini_cluster.cc:1428] Setting key 2c3e78ca3c98a2c0fb2702ab1b23ca89
I20260929 03:14:36.772568 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.764396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.67:34327 (local address 127.20.174.126:40341)
0929 03:14:36.764555 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:36.764560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:36.765785 (+ 1225us) server_negotiation.cc:407] Connection header received
0929 03:14:36.767255 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:36.767257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:36.767307 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:36.767386 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:36.768399 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.768944 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.770174 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.770552 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.770643 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.772035 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:36.772038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:36.772041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:36.772061 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:36.772081 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:36.772089 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:36.772158 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:36.772356 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 03:14:36.772432 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:14:36.773603 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.764899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:40341 (local address 127.20.174.67:34327)
0929 03:14:36.765612 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:36.765654 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:36.767032 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:36.767588 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:36.767595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:36.767703 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:14:36.768242 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.768261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.769043 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.769046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.770042 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.770052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.770701 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.770705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.770855 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.771669 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:14:36.771672 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:14:36.771735 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:14:36.771824 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:14:36.772190 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:14:36.772205 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:14:36.772210 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:14:36.772318 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:14:36.772663 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":137,"threads_started":1}
I20260929 03:14:36.774897 8916 heartbeater.cc:347] Connected to a master server at 127.20.174.126:40341
I20260929 03:14:36.775126 8916 heartbeater.cc:464] Registering TS with master...
I20260929 03:14:36.775619 8916 heartbeater.cc:511] Master 127.20.174.126:40341 requested a full tablet report, sending...
I20260929 03:14:36.776624 8465 ts_manager.cc:194] Registered new tserver with Master: f185c62a0c4a47e0851f6a8fa16d744f (127.20.174.67:39791)
I20260929 03:14:36.777204 8465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.174.67:34327
I20260929 03:14:36.786623 21177 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:14:36.793082 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.788739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.20.174.126:40341)
0929 03:14:36.788973 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:36.788978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:36.789122 (+ 144us) server_negotiation.cc:407] Connection header received
0929 03:14:36.789301 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:36.789304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:36.789353 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:36.789441 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:36.789804 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.790337 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.791273 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.791537 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.791652 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.792609 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:36.792614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:36.792620 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:36.792638 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:36.792662 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:36.792672 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:36.792754 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:36.792905 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 03:14:36.792963 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 03:14:36.796429 8465 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:14:36.799626 8465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:14:36.815599 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.809180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.66:46125 (local address 127.0.0.1:46118)
0929 03:14:36.809743 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:36.809799 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:36.809947 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:36.810904 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:36.810912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:36.810938 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:36.811078 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.811083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.812827 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.812832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.814074 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.814084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.815123 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.815126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.815214 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.815245 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:14:36.815333 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 03:14:36.815435 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":101,"thread_start_us":66,"threads_started":1}
I20260929 03:14:36.815747 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.809399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.20.174.66:46125)
0929 03:14:36.810253 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:36.810326 (+ 73us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:36.810353 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:14:36.810462 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:36.810468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:36.810701 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:36.811233 (+ 532us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:36.811389 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.812680 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.814219 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.815001 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.815111 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.815320 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:36.815374 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 03:14:36.815475 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":138,"threads_started":1}
I20260929 03:14:36.816229 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.809950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:43543 (local address 127.0.0.1:38362)
0929 03:14:36.810759 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:36.810787 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:36.810871 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:36.811501 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:36.811504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:36.811524 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:36.811637 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.811642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.812744 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.812747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.814504 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.814509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.815621 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.815627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.815778 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.815801 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:14:36.816015 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 03:14:36.816094 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":128,"thread_start_us":96,"threads_started":1}
I20260929 03:14:36.817567 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.809451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:39791 (local address 127.0.0.1:40418)
0929 03:14:36.809832 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:36.809849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:36.809947 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:36.811275 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:36.811278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:36.811293 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:36.811389 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.811395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.812840 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.812842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.814498 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.814509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.817067 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.817070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.817170 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.817190 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:14:36.817264 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:14:36.817349 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":171,"thread_start_us":103,"threads_started":1}
I20260929 03:14:36.817601 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.810378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.20.174.65:43543)
0929 03:14:36.810969 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:36.810976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:36.810996 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:14:36.811072 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:36.811076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:36.811242 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:36.811745 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:36.811872 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.812607 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.814648 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.815486 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.815606 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.817276 (+ 1670us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:36.817330 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 03:14:36.817426 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":113,"threads_started":1}
I20260929 03:14:36.818074 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.810404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.20.174.67:39791)
0929 03:14:36.810776 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:36.810789 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:36.810834 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:14:36.810911 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:36.810916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:36.811112 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:36.811278 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:36.811529 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.812711 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.816085 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.816816 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.816929 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.817105 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:36.817784 (+ 679us) server_negotiation.cc:299] Negotiation successful
0929 03:14:36.817901 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":170,"threads_started":1}
I20260929 03:14:36.819306 8719 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc12e7e189c4678b4087175c5a417c1 (DEFAULT_TABLE table=test-table [id=43828008d2aa470ea38baf8c320f75f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:36.820405 8719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc12e7e189c4678b4087175c5a417c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:36.820588 8588 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc12e7e189c4678b4087175c5a417c1 (DEFAULT_TABLE table=test-table [id=43828008d2aa470ea38baf8c320f75f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:36.820597 8850 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc12e7e189c4678b4087175c5a417c1 (DEFAULT_TABLE table=test-table [id=43828008d2aa470ea38baf8c320f75f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:14:36.821687 8588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc12e7e189c4678b4087175c5a417c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:36.821687 8850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc12e7e189c4678b4087175c5a417c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:36.827343 8933 tablet_bootstrap.cc:492] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45: Bootstrap starting.
I20260929 03:14:36.829074 8934 tablet_bootstrap.cc:492] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba: Bootstrap starting.
I20260929 03:14:36.830008 8933 tablet_bootstrap.cc:654] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:36.830768 8935 tablet_bootstrap.cc:492] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f: Bootstrap starting.
I20260929 03:14:36.830965 8933 log.cc:824] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:36.831403 8934 tablet_bootstrap.cc:654] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:36.832273 8934 log.cc:824] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:36.832966 8935 tablet_bootstrap.cc:654] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:36.833465 8933 tablet_bootstrap.cc:492] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45: No bootstrap required, opened a new log
I20260929 03:14:36.833663 8933 ts_tablet_manager.cc:1397] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:14:36.833693 8935 log.cc:824] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:36.834456 8934 tablet_bootstrap.cc:492] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba: No bootstrap required, opened a new log
I20260929 03:14:36.834643 8934 ts_tablet_manager.cc:1397] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:14:36.835381 8935 tablet_bootstrap.cc:492] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f: No bootstrap required, opened a new log
I20260929 03:14:36.835577 8935 ts_tablet_manager.cc:1397] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 03:14:36.838222 8933 raft_consensus.cc:348] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 46125 } } peers { permanent_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39791 } } peers { permanent_uuid: "ef4f66ce9269455fa680560500beffba" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43543 } }
I20260929 03:14:36.838532 8933 raft_consensus.cc:374] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:36.838587 8933 raft_consensus.cc:729] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f59d7c35be2434aaf9b9544f7aaec45, State: Initialized, Role: FOLLOWER
I20260929 03:14:36.838661 8935 raft_consensus.cc:348] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 46125 } } peers { permanent_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39791 } } peers { permanent_uuid: "ef4f66ce9269455fa680560500beffba" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43543 } }
I20260929 03:14:36.838904 8935 raft_consensus.cc:374] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:36.838965 8935 raft_consensus.cc:729] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f185c62a0c4a47e0851f6a8fa16d744f, State: Initialized, Role: FOLLOWER
I20260929 03:14:36.838945 8934 raft_consensus.cc:348] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 46125 } } peers { permanent_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39791 } } peers { permanent_uuid: "ef4f66ce9269455fa680560500beffba" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43543 } }
I20260929 03:14:36.839068 8933 consensus_queue.cc:260] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 46125 } } peers { permanent_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39791 } } peers { permanent_uuid: "ef4f66ce9269455fa680560500beffba" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43543 } }
I20260929 03:14:36.839186 8934 raft_consensus.cc:374] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:36.839229 8934 raft_consensus.cc:729] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4f66ce9269455fa680560500beffba, State: Initialized, Role: FOLLOWER
I20260929 03:14:36.839413 8935 consensus_queue.cc:260] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 46125 } } peers { permanent_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39791 } } peers { permanent_uuid: "ef4f66ce9269455fa680560500beffba" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43543 } }
I20260929 03:14:36.839730 8934 consensus_queue.cc:260] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 46125 } } peers { permanent_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39791 } } peers { permanent_uuid: "ef4f66ce9269455fa680560500beffba" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43543 } }
I20260929 03:14:36.840145 8933 ts_tablet_manager.cc:1428] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:14:36.840364 8935 ts_tablet_manager.cc:1428] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260929 03:14:36.840453 8654 heartbeater.cc:503] Master 127.20.174.126:40341 was elected leader, sending a full tablet report...
I20260929 03:14:36.840513 8785 heartbeater.cc:503] Master 127.20.174.126:40341 was elected leader, sending a full tablet report...
I20260929 03:14:36.840637 8916 heartbeater.cc:503] Master 127.20.174.126:40341 was elected leader, sending a full tablet report...
I20260929 03:14:36.840684 8934 ts_tablet_manager.cc:1428] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:14:36.932771 8939 raft_consensus.cc:482] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:14:36.932971 8939 raft_consensus.cc:504] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 46125 } } peers { permanent_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39791 } } peers { permanent_uuid: "ef4f66ce9269455fa680560500beffba" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43543 } }
I20260929 03:14:36.934151 8939 leader_election.cc:288] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f185c62a0c4a47e0851f6a8fa16d744f (127.20.174.67:39791), ef4f66ce9269455fa680560500beffba (127.20.174.65:43543)
I20260929 03:14:36.938020 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.934600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:53479 (local address 127.20.174.65:43543)
0929 03:14:36.934812 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:36.934817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:36.934834 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:14:36.934894 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:36.934897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:36.934960 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:36.935053 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:36.935399 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.935920 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.937035 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.937442 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.937579 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.937630 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:36.937820 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 03:14:36.937891 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260929 03:14:36.938020 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.934415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:43543 (local address 127.20.174.66:53479)
0929 03:14:36.934586 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:36.934606 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:36.934708 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:36.935069 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:36.935072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:36.935090 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:36.935251 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.935258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.936043 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.936049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.936897 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.936906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.937573 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.937577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.937682 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.937709 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 03:14:36.937802 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:14:36.937880 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":27}
I20260929 03:14:36.938778 8942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.934411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.67:39791 (local address 127.20.174.66:34709)
0929 03:14:36.935096 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:36.935109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:14:36.935245 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:14:36.935544 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:14:36.935547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:14:36.935564 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:14:36.935699 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.935705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.936495 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.936500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.937420 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:14:36.937430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.938341 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.938345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:14:36.938480 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.938503 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:14:36.938591 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 03:14:36.938659 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":129,"thread_start_us":290,"threads_started":1}
I20260929 03:14:36.938856 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.934868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.174.66:34709 (local address 127.20.174.67:39791)
0929 03:14:36.934997 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:36.935001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:36.935208 (+ 207us) server_negotiation.cc:407] Connection header received
0929 03:14:36.935374 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:36.935377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:36.935437 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:36.935532 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:14:36.935820 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.936369 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.937562 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.938031 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.938204 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.938270 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:36.938647 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 03:14:36.938741 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:14:36.938928 8608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc12e7e189c4678b4087175c5a417c1" candidate_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef4f66ce9269455fa680560500beffba" is_pre_election: true
I20260929 03:14:36.939280 8608 raft_consensus.cc:2454] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f59d7c35be2434aaf9b9544f7aaec45 in term 0.
I20260929 03:14:36.939616 8870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc12e7e189c4678b4087175c5a417c1" candidate_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" is_pre_election: true
I20260929 03:14:36.939924 8870 raft_consensus.cc:2454] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f59d7c35be2434aaf9b9544f7aaec45 in term 0.
I20260929 03:14:36.939906 8673 leader_election.cc:302] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f59d7c35be2434aaf9b9544f7aaec45, ef4f66ce9269455fa680560500beffba; no voters:
I20260929 03:14:36.940235 8939 raft_consensus.cc:2781] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:14:36.940409 8939 raft_consensus.cc:482] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:14:36.940464 8939 raft_consensus.cc:3037] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:36.941607 8939 raft_consensus.cc:504] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 46125 } } peers { permanent_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39791 } } peers { permanent_uuid: "ef4f66ce9269455fa680560500beffba" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43543 } }
I20260929 03:14:36.942026 8939 leader_election.cc:288] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [CANDIDATE]: Term 1 election: Requested vote from peers f185c62a0c4a47e0851f6a8fa16d744f (127.20.174.67:39791), ef4f66ce9269455fa680560500beffba (127.20.174.65:43543)
I20260929 03:14:36.942449 8870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc12e7e189c4678b4087175c5a417c1" candidate_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f185c62a0c4a47e0851f6a8fa16d744f"
I20260929 03:14:36.942593 8608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc12e7e189c4678b4087175c5a417c1" candidate_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef4f66ce9269455fa680560500beffba"
I20260929 03:14:36.942600 8870 raft_consensus.cc:3037] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:36.942848 8608 raft_consensus.cc:3037] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:36.943861 8870 raft_consensus.cc:2454] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f59d7c35be2434aaf9b9544f7aaec45 in term 1.
I20260929 03:14:36.944149 8608 raft_consensus.cc:2454] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f59d7c35be2434aaf9b9544f7aaec45 in term 1.
I20260929 03:14:36.944245 8674 leader_election.cc:302] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f59d7c35be2434aaf9b9544f7aaec45, f185c62a0c4a47e0851f6a8fa16d744f; no voters:
I20260929 03:14:36.944434 8939 raft_consensus.cc:2781] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:36.944653 8939 raft_consensus.cc:686] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 1 LEADER]: Becoming Leader. State: Replica: 8f59d7c35be2434aaf9b9544f7aaec45, State: Running, Role: LEADER
I20260929 03:14:36.945123 8939 consensus_queue.cc:237] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 46125 } } peers { permanent_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39791 } } peers { permanent_uuid: "ef4f66ce9269455fa680560500beffba" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43543 } }
I20260929 03:14:36.948742 8466 catalog_manager.cc:5495] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f59d7c35be2434aaf9b9544f7aaec45 (127.20.174.66). New cstate: current_term: 1 leader_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f59d7c35be2434aaf9b9544f7aaec45" member_type: VOTER last_known_addr { host: "127.20.174.66" port: 46125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef4f66ce9269455fa680560500beffba" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43543 } health_report { overall_health: UNKNOWN } } }
I20260929 03:14:36.970362 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:36.966752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.20.174.66:46125)
0929 03:14:36.966917 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:36.966922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:36.966936 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:14:36.967058 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:36.967060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:36.967116 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:36.967260 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:14:36.967520 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.968041 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.968892 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:36.969120 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:36.969241 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:36.969543 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:14:36.969970 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:14:36.970111 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:36.970151 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 03:14:36.970213 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:14:36.976678 8870 raft_consensus.cc:1261] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f [term 1 FOLLOWER]: Refusing update from remote peer 8f59d7c35be2434aaf9b9544f7aaec45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:36.976678 8608 raft_consensus.cc:1261] T 9fc12e7e189c4678b4087175c5a417c1 P ef4f66ce9269455fa680560500beffba [term 1 FOLLOWER]: Refusing update from remote peer 8f59d7c35be2434aaf9b9544f7aaec45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:14:36.977502 8943 consensus_queue.cc:1035] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef4f66ce9269455fa680560500beffba" member_type: VOTER last_known_addr { host: "127.20.174.65" port: 43543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:36.977763 8939 consensus_queue.cc:1035] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f185c62a0c4a47e0851f6a8fa16d744f" member_type: VOTER last_known_addr { host: "127.20.174.67" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:14:36.984545 8946 mvcc.cc:205] Tried to move back new op lower bound from 7334509268888449024 to 7334509268770074624. Current Snapshot: MvccSnapshot[applied={T|T < 7334509268888449024}]
I20260929 03:14:36.988710 8947 mvcc.cc:205] Tried to move back new op lower bound from 7334509268888449024 to 7334509268770074624. Current Snapshot: MvccSnapshot[applied={T|T < 7334509268888449024}]
I20260929 03:14:36.994709 8463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:36.994891 8463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:14:36.997648 8463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 978d6d0364a2447bb10e9d3e762c80ca: Sending DeleteTablet for 3 replicas of tablet 9fc12e7e189c4678b4087175c5a417c1
I20260929 03:14:36.998593 8850 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc12e7e189c4678b4087175c5a417c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:36 UTC) from {username='slave'} at 127.0.0.1:40418
I20260929 03:14:36.998659 8588 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc12e7e189c4678b4087175c5a417c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:36 UTC) from {username='slave'} at 127.0.0.1:38362
I20260929 03:14:36.999596 8719 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc12e7e189c4678b4087175c5a417c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:14:36 UTC) from {username='slave'} at 127.0.0.1:46118
I20260929 03:14:37.000203 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 8526
I20260929 03:14:37.001317 8958 tablet_replica.cc:330] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45: stopping tablet replica
I20260929 03:14:37.001642 8958 raft_consensus.cc:2229] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:14:37.002039 8958 raft_consensus.cc:2258] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:37.003693 8958 ts_tablet_manager.cc:1905] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:14:37.006559 8958 ts_tablet_manager.cc:1918] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:37.006659 8958 log.cc:1196] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-1/wal/wals/9fc12e7e189c4678b4087175c5a417c1
I20260929 03:14:37.006958 8958 ts_tablet_manager.cc:1939] T 9fc12e7e189c4678b4087175c5a417c1 P 8f59d7c35be2434aaf9b9544f7aaec45: Deleting consensus metadata
I20260929 03:14:37.008350 8453 catalog_manager.cc:4841] TS 8f59d7c35be2434aaf9b9544f7aaec45 (127.20.174.66:46125): tablet 9fc12e7e189c4678b4087175c5a417c1 (table test-table [id=43828008d2aa470ea38baf8c320f75f0]) successfully deleted
I20260929 03:14:37.006964 8956 tablet_replica.cc:330] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f: stopping tablet replica
I20260929 03:14:37.008994 8956 raft_consensus.cc:2229] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:14:37.009349 8956 raft_consensus.cc:2258] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:14:37.010890 8956 ts_tablet_manager.cc:1905] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:14:37.012743 8451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.174.65:43543 (error 108)
I20260929 03:14:37.013546 8956 ts_tablet_manager.cc:1918] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:14:37.013648 8956 log.cc:1196] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f: Deleting WAL directory at /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651413504048-21177-0/minicluster-data/ts-2/wal/wals/9fc12e7e189c4678b4087175c5a417c1
I20260929 03:14:37.013631 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 8657
I20260929 03:14:37.014025 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:37.013586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.65:43543 (local address 127.0.0.1:38378)
0929 03:14:37.013852 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 03:14:37.013929 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.65:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 03:14:37.014681 8956 ts_tablet_manager.cc:1939] T 9fc12e7e189c4678b4087175c5a417c1 P f185c62a0c4a47e0851f6a8fa16d744f: Deleting consensus metadata
W20260929 03:14:37.014724 8451 catalog_manager.cc:4584] TS ef4f66ce9269455fa680560500beffba (127.20.174.65:43543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fc12e7e189c4678b4087175c5a417c1: Network error: Client connection negotiation failed: client connection to 127.20.174.65:43543: connect: Connection refused (error 111)
I20260929 03:14:37.016032 8452 catalog_manager.cc:4841] TS f185c62a0c4a47e0851f6a8fa16d744f (127.20.174.67:39791): tablet 9fc12e7e189c4678b4087175c5a417c1 (table test-table [id=43828008d2aa470ea38baf8c320f75f0]) successfully deleted
I20260929 03:14:37.020623 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 8788
I20260929 03:14:37.025969 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 8435
2026-09-29T03:14:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1251 ms)
[----------] 6 tests from EncryptionPolicyTest (11149 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:37 dist-test-slave-k1dg krb5kdc[8962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:37 dist-test-slave-k1dg krb5kdc[8962](info): set up 2 sockets
Sep 29 03:14:37 dist-test-slave-k1dg krb5kdc[8962](info): commencing operation
krb5kdc: starting...
W20260929 03:14:38.086906 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:38 dist-test-slave-k1dg krb5kdc[8962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:38.223183 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:38.343060 8975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:38.343391 8975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:38.343469 8975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:38.347831 8975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:38.347913 8975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:38.347934 8975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:38.347951 8975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:38.348014 8975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:38.353822 8975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:38.355154 8975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:38.356276 8975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:38.365406 8980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:38.365393 8981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:38.365453 8983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:38.365784 8975 server_base.cc:1034] running on GCE node
I20260929 03:14:38.366652 8975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:38.367954 8975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:38.369127 8975 hybrid_clock.cc:648] HybridClock initialized: now 1790651678369095 us; error 55 us; skew 500 ppm
Sep 29 03:14:38 dist-test-slave-k1dg krb5kdc[8962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:38.372176 8975 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:38.373435 8975 webserver.cc:460] Webserver started at http://127.0.0.1:37411/ using document root <none> and password file <none>
I20260929 03:14:38.374039 8975 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:38.374104 8975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:38.374368 8975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:38.376605 8975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "e5ef8943853f407383426ef3dfbafc7c"
format_stamp: "Formatted at 2026-09-29 03:14:38 on dist-test-slave-k1dg"
server_key: "65e697bbcd33970c9093f696c34a330e"
server_key_iv: "80feb96b75a2b9f5dc667f808e9bbb8e"
server_key_version: "encryptionkey@0"
I20260929 03:14:38.377230 8975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "e5ef8943853f407383426ef3dfbafc7c"
format_stamp: "Formatted at 2026-09-29 03:14:38 on dist-test-slave-k1dg"
server_key: "65e697bbcd33970c9093f696c34a330e"
server_key_iv: "80feb96b75a2b9f5dc667f808e9bbb8e"
server_key_version: "encryptionkey@0"
I20260929 03:14:38.381556 8975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:14:38.384543 8990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:38.386307 8975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:14:38.386549 8975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "e5ef8943853f407383426ef3dfbafc7c"
format_stamp: "Formatted at 2026-09-29 03:14:38 on dist-test-slave-k1dg"
server_key: "65e697bbcd33970c9093f696c34a330e"
server_key_iv: "80feb96b75a2b9f5dc667f808e9bbb8e"
server_key_version: "encryptionkey@0"
I20260929 03:14:38.386812 8975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:38.404619 8975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:38.408102 8975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:38.408345 8975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:38.416244 8975 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35801
I20260929 03:14:38.416272 9042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35801 every 8 connection(s)
I20260929 03:14:38.417248 8975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:38.420347 9043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:38.420482 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 8975
I20260929 03:14:38.420624 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:38.420948 21177 external_mini_cluster.cc:1428] Setting key 4fccbd91e719bd26bab9dcbce9601924
I20260929 03:14:38.427246 9043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c: Bootstrap starting.
I20260929 03:14:38.429649 9043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c: Neither blocks nor log segments found. Creating new log.
Sep 29 03:14:38 dist-test-slave-k1dg krb5kdc[8962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:14:38.430639 9043 log.cc:824] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:38.433033 9043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c: No bootstrap required, opened a new log
I20260929 03:14:38.436523 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:38.422539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44793 (local address 127.0.0.1:35801)
0929 03:14:38.422986 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:38.423002 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:38.423037 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:14:38.423835 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:38.423856 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:38.424218 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:38.424631 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:38.425514 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:38.426609 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:38.427501 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:38.428000 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:38.428178 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:38.431081 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:38.431109 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:38.431130 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:38.431167 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:38.433213 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:38.433827 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:38.433832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:38.433837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:38.433976 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:38.434345 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:38.434347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:38.434348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:38.434865 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:38.435037 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:38.435425 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 03:14:38.435705 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":126,"threads_started":1}
I20260929 03:14:38.437520 9043 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef8943853f407383426ef3dfbafc7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } }
I20260929 03:14:38.437853 9043 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:38.437909 9043 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ef8943853f407383426ef3dfbafc7c, State: Initialized, Role: FOLLOWER
I20260929 03:14:38.438396 9043 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef8943853f407383426ef3dfbafc7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } }
I20260929 03:14:38.438613 9043 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:38.438681 9043 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:38.438789 9043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:38.439924 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:38.440065 9043 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef8943853f407383426ef3dfbafc7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } }
I20260929 03:14:38.440481 9043 leader_election.cc:302] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5ef8943853f407383426ef3dfbafc7c; no voters:
I20260929 03:14:38.440897 9043 leader_election.cc:288] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:38.441033 9048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:38.441318 9048 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [term 1 LEADER]: Becoming Leader. State: Replica: e5ef8943853f407383426ef3dfbafc7c, State: Running, Role: LEADER
I20260929 03:14:38.441689 9048 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef8943853f407383426ef3dfbafc7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } }
I20260929 03:14:38.442457 9043 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:38.443120 9048 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5ef8943853f407383426ef3dfbafc7c. Latest consensus state: current_term: 1 leader_uuid: "e5ef8943853f407383426ef3dfbafc7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef8943853f407383426ef3dfbafc7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } } }
I20260929 03:14:38.443255 9048 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:38.443572 9050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5ef8943853f407383426ef3dfbafc7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef8943853f407383426ef3dfbafc7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } } }
I20260929 03:14:38.443665 9050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:38.443813 9060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:38.447738 9060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:38.452929 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:38.442360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59569 (local address 127.0.0.1:35801)
0929 03:14:38.442504 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:38.442509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:38.442767 (+ 258us) server_negotiation.cc:407] Connection header received
0929 03:14:38.442990 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:38.442993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:38.443044 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:38.443096 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:38.444154 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:38.445022 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:38.446275 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:38.446613 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:38.446734 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:38.448567 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:38.448570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:38.448573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:38.448601 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:38.450345 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:38.451078 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:38.451081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:38.451083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:38.451182 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:38.451618 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:38.451621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:38.451623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:38.451868 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:38.452013 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:38.452393 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 03:14:38.452746 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:14:38.455087 9060 catalog_manager.cc:1347] Generated new cluster ID: 537f86603e0b4bb581dd6062fd04b1e0
I20260929 03:14:38.455180 9060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:38.474272 9060 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:38.475526 9060 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:38.483732 9060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5ef8943853f407383426ef3dfbafc7c: Generated new TSK 0
I20260929 03:14:38.484759 9060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:38.499809 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 8975
W20260929 03:14:38.505616 8960 connection.cc:535] client connection to 127.0.0.1:35801 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35801): Connection reset by peer (error 104)
2026-09-29T03:14:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:38 dist-test-slave-k1dg krb5kdc[9074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:38 dist-test-slave-k1dg krb5kdc[9074](info): set up 2 sockets
Sep 29 03:14:38 dist-test-slave-k1dg krb5kdc[9074](info): commencing operation
krb5kdc: starting...
W20260929 03:14:39.555251 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:39 dist-test-slave-k1dg krb5kdc[9074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:39.691555 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:39.809123 9087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:39.809460 9087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:39.809552 9087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:39.813896 9087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:39.814006 9087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:39.814047 9087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:39.814085 9087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:39.814123 9087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:39.820012 9087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:39.821272 9087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:39.822364 9087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:39.832257 9093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:39.832260 9092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:39.832324 9087 server_base.cc:1034] running on GCE node
W20260929 03:14:39.832260 9095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:39.833153 9087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:39.834373 9087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:39.835749 9087 hybrid_clock.cc:648] HybridClock initialized: now 1790651679835610 us; error 57 us; skew 500 ppm
Sep 29 03:14:39 dist-test-slave-k1dg krb5kdc[9074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:39.839215 9087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:39.840281 9087 webserver.cc:460] Webserver started at http://127.0.0.1:42263/ using document root <none> and password file <none>
I20260929 03:14:39.840932 9087 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:39.841037 9087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:39.841302 9087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:39.843741 9087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "69c11a0ea7194623a0a45919cabe7162"
format_stamp: "Formatted at 2026-09-29 03:14:39 on dist-test-slave-k1dg"
server_key: "5b2e94195908ba8bbcdbe8edfe0bf777"
server_key_iv: "1c4924103b54f8d9dfe95f3b17e4e4b6"
server_key_version: "encryptionkey@0"
I20260929 03:14:39.844412 9087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "69c11a0ea7194623a0a45919cabe7162"
format_stamp: "Formatted at 2026-09-29 03:14:39 on dist-test-slave-k1dg"
server_key: "5b2e94195908ba8bbcdbe8edfe0bf777"
server_key_iv: "1c4924103b54f8d9dfe95f3b17e4e4b6"
server_key_version: "encryptionkey@0"
I20260929 03:14:39.849512 9087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:14:39.852567 9102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:39.854025 9087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:14:39.854192 9087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "69c11a0ea7194623a0a45919cabe7162"
format_stamp: "Formatted at 2026-09-29 03:14:39 on dist-test-slave-k1dg"
server_key: "5b2e94195908ba8bbcdbe8edfe0bf777"
server_key_iv: "1c4924103b54f8d9dfe95f3b17e4e4b6"
server_key_version: "encryptionkey@0"
I20260929 03:14:39.854326 9087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:39.873687 9087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:39.877616 9087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:39.877875 9087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:39.885531 9087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39971
I20260929 03:14:39.885545 9154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39971 every 8 connection(s)
I20260929 03:14:39.886652 9087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:39.889263 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 9087
I20260929 03:14:39.889425 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:39.889760 21177 external_mini_cluster.cc:1428] Setting key 7104be33732290a196f1c2c7d421dd5d
I20260929 03:14:39.890064 9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:39.897025 9155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162: Bootstrap starting.
I20260929 03:14:39.899663 9155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:39.899963 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:39.891373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55387 (local address 127.0.0.1:39971)
0929 03:14:39.891907 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:39.891922 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:39.891961 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:14:39.893026 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:39.893047 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:39.893427 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:39.893806 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:39.894265 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:39.895379 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:39.896330 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:39.896857 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:39.897036 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:39.898065 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:39.898070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:39.898084 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:39.898563 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:39.898621 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:39.898646 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:39.898801 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:39.898989 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 03:14:39.899142 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":145,"threads_started":1}
I20260929 03:14:39.900589 9155 log.cc:824] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:39.902835 9155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162: No bootstrap required, opened a new log
I20260929 03:14:39.903069 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:39.906795 9155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c11a0ea7194623a0a45919cabe7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39971 } }
I20260929 03:14:39.907061 9155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:39.907146 9155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c11a0ea7194623a0a45919cabe7162, State: Initialized, Role: FOLLOWER
I20260929 03:14:39.907675 9155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c11a0ea7194623a0a45919cabe7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39971 } }
I20260929 03:14:39.907840 9155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:39.907930 9155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:39.908034 9155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:39.909019 9155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c11a0ea7194623a0a45919cabe7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39971 } }
I20260929 03:14:39.909423 9155 leader_election.cc:302] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c11a0ea7194623a0a45919cabe7162; no voters:
I20260929 03:14:39.909574 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:39.905089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34353 (local address 127.0.0.1:39971)
0929 03:14:39.905207 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:39.905211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:39.905374 (+ 163us) server_negotiation.cc:407] Connection header received
0929 03:14:39.905516 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:39.905519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:39.905564 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:39.905652 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:39.906003 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:39.906574 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:39.907552 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:39.907803 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:39.907947 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:39.909045 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:39.909048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:39.909051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:39.909064 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:39.909097 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:39.909108 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:39.909206 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:39.909345 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 03:14:39.909377 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:14:39.909852 9155 leader_election.cc:288] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:39.909948 9165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:39.910166 9165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [term 1 LEADER]: Becoming Leader. State: Replica: 69c11a0ea7194623a0a45919cabe7162, State: Running, Role: LEADER
I20260929 03:14:39.910545 9165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c11a0ea7194623a0a45919cabe7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39971 } }
I20260929 03:14:39.911180 9155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:39.912220 9166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c11a0ea7194623a0a45919cabe7162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c11a0ea7194623a0a45919cabe7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39971 } } }
I20260929 03:14:39.912349 9166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:39.912664 9167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c11a0ea7194623a0a45919cabe7162. Latest consensus state: current_term: 1 leader_uuid: "69c11a0ea7194623a0a45919cabe7162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c11a0ea7194623a0a45919cabe7162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39971 } } }
I20260929 03:14:39.912761 9174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:39.912784 9167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:39.916031 9174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:39.922353 9174 catalog_manager.cc:1347] Generated new cluster ID: 637dfb541dbb4497beadcd2c0f1a283c
I20260929 03:14:39.922537 9174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:39.935570 9174 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:39.936611 9174 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:39.941512 9174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69c11a0ea7194623a0a45919cabe7162: Generated new TSK 0
I20260929 03:14:39.942917 9174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:39.958056 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 9087
W20260929 03:14:39.963496 9072 connection.cc:535] client connection to 127.0.0.1:39971 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39971): Connection reset by peer (error 104)
2026-09-29T03:14:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:40 dist-test-slave-k1dg krb5kdc[9186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:40 dist-test-slave-k1dg krb5kdc[9186](info): set up 2 sockets
Sep 29 03:14:40 dist-test-slave-k1dg krb5kdc[9186](info): commencing operation
krb5kdc: starting...
W20260929 03:14:41.010923 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:41 dist-test-slave-k1dg krb5kdc[9186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:41.138576 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:41.253047 9199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:41.253412 9199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:41.253492 9199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:41.257761 9199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:41.257879 9199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:41.257915 9199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:41.257957 9199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:41.257987 9199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:41.263602 9199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37451
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:41.264868 9199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:41.265966 9199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:41.275257 9205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:41.275311 9199 server_base.cc:1034] running on GCE node
W20260929 03:14:41.275328 9204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:41.275501 9207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:41.276238 9199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:41.277451 9199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:41.278709 9199 hybrid_clock.cc:648] HybridClock initialized: now 1790651681278642 us; error 82 us; skew 500 ppm
Sep 29 03:14:41 dist-test-slave-k1dg krb5kdc[9186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:41.291683 9199 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:41.292838 9199 webserver.cc:460] Webserver started at http://127.0.0.1:39573/ using document root <none> and password file <none>
I20260929 03:14:41.293452 9199 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:41.293550 9199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:41.293807 9199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:41.296062 9199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "b78299fc2b0342e6a8aad94fd0bab9c5"
format_stamp: "Formatted at 2026-09-29 03:14:41 on dist-test-slave-k1dg"
server_key: "632e660c7bb9ccd99cbc091c734dcff1"
server_key_iv: "161b574cb8076ae2c46efeb6eefbf859"
server_key_version: "encryptionkey@0"
I20260929 03:14:41.296679 9199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "b78299fc2b0342e6a8aad94fd0bab9c5"
format_stamp: "Formatted at 2026-09-29 03:14:41 on dist-test-slave-k1dg"
server_key: "632e660c7bb9ccd99cbc091c734dcff1"
server_key_iv: "161b574cb8076ae2c46efeb6eefbf859"
server_key_version: "encryptionkey@0"
I20260929 03:14:41.300680 9199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:14:41.303422 9214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:41.304731 9199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:41.304886 9199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "b78299fc2b0342e6a8aad94fd0bab9c5"
format_stamp: "Formatted at 2026-09-29 03:14:41 on dist-test-slave-k1dg"
server_key: "632e660c7bb9ccd99cbc091c734dcff1"
server_key_iv: "161b574cb8076ae2c46efeb6eefbf859"
server_key_version: "encryptionkey@0"
I20260929 03:14:41.305009 9199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:41.310114 9199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:41.313246 9199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:41.313470 9199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:41.321178 9199 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37451
I20260929 03:14:41.321209 9266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37451 every 8 connection(s)
I20260929 03:14:41.322329 9199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:41.325294 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 9199
I20260929 03:14:41.325480 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:41.325654 9267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:41.325858 21177 external_mini_cluster.cc:1428] Setting key 49044c265193e6f3b69623365967e5db
I20260929 03:14:41.331553 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:41.327278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43437 (local address 127.0.0.1:37451)
0929 03:14:41.327732 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:41.327749 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:41.327789 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:14:41.328551 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:41.328571 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:41.328877 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:41.329173 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:41.330171 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:41.330175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:41.330187 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:41.330215 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:41.330274 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:41.330294 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:41.330409 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:41.330580 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 03:14:41.330754 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":149,"threads_started":1}
I20260929 03:14:41.333354 9267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5: Bootstrap starting.
I20260929 03:14:41.334661 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:41.335949 9267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:41.336761 9267 log.cc:824] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:41.338850 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:41.336842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51857 (local address 127.0.0.1:37451)
0929 03:14:41.337036 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:41.337041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:41.337273 (+ 232us) server_negotiation.cc:407] Connection header received
0929 03:14:41.337446 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:41.337453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:41.337486 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:41.337567 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:41.338450 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:41.338454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:41.338457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:41.338470 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:41.338499 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:41.338510 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:41.338604 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:41.338676 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:14:41.338718 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 03:14:41.339082 9267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5: No bootstrap required, opened a new log
I20260929 03:14:41.342845 9267 raft_consensus.cc:348] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78299fc2b0342e6a8aad94fd0bab9c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } }
I20260929 03:14:41.343106 9267 raft_consensus.cc:374] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:41.343168 9267 raft_consensus.cc:729] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b78299fc2b0342e6a8aad94fd0bab9c5, State: Initialized, Role: FOLLOWER
I20260929 03:14:41.343665 9267 consensus_queue.cc:260] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78299fc2b0342e6a8aad94fd0bab9c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } }
I20260929 03:14:41.343815 9267 raft_consensus.cc:388] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:41.343900 9267 raft_consensus.cc:482] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:41.344025 9267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:41.345021 9267 raft_consensus.cc:504] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78299fc2b0342e6a8aad94fd0bab9c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } }
I20260929 03:14:41.345378 9267 leader_election.cc:302] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b78299fc2b0342e6a8aad94fd0bab9c5; no voters:
I20260929 03:14:41.345778 9267 leader_election.cc:288] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:41.345858 9277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:41.346065 9277 raft_consensus.cc:686] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [term 1 LEADER]: Becoming Leader. State: Replica: b78299fc2b0342e6a8aad94fd0bab9c5, State: Running, Role: LEADER
I20260929 03:14:41.346475 9277 consensus_queue.cc:237] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78299fc2b0342e6a8aad94fd0bab9c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } }
I20260929 03:14:41.347270 9267 sys_catalog.cc:564] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:41.348387 9279 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b78299fc2b0342e6a8aad94fd0bab9c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78299fc2b0342e6a8aad94fd0bab9c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } } }
I20260929 03:14:41.348519 9279 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:41.348820 9278 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b78299fc2b0342e6a8aad94fd0bab9c5. Latest consensus state: current_term: 1 leader_uuid: "b78299fc2b0342e6a8aad94fd0bab9c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78299fc2b0342e6a8aad94fd0bab9c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37451 } } }
I20260929 03:14:41.348904 9278 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:41.348955 9286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:41.352471 9286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:41.358776 9286 catalog_manager.cc:1347] Generated new cluster ID: 85ff6a97cf05428b996e1d6bf6519498
I20260929 03:14:41.358867 9286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:41.387231 9286 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:41.387601 9286 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:41.397459 9286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b78299fc2b0342e6a8aad94fd0bab9c5: Generated new TSK 0
I20260929 03:14:41.398379 9286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:41.402806 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 9199
2026-09-29T03:14:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:41 dist-test-slave-k1dg krb5kdc[9298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:41 dist-test-slave-k1dg krb5kdc[9298](info): set up 2 sockets
Sep 29 03:14:41 dist-test-slave-k1dg krb5kdc[9298](info): commencing operation
krb5kdc: starting...
W20260929 03:14:42.475310 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:42 dist-test-slave-k1dg krb5kdc[9298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:42.619014 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:42.738677 9311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:42.738996 9311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:42.739040 9311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:42.743450 9311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:42.743541 9311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:42.743561 9311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:42.743578 9311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:42.743594 9311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:42.749493 9311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:42.750752 9311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:42.751892 9311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:42.761431 9316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:42.761579 9319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:42.761417 9317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:42.761600 9311 server_base.cc:1034] running on GCE node
I20260929 03:14:42.762527 9311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:42.763803 9311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:42.765024 9311 hybrid_clock.cc:648] HybridClock initialized: now 1790651682764986 us; error 54 us; skew 500 ppm
Sep 29 03:14:42 dist-test-slave-k1dg krb5kdc[9298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:42.768414 9311 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:42.769778 9311 webserver.cc:460] Webserver started at http://127.0.0.1:33103/ using document root <none> and password file <none>
I20260929 03:14:42.770532 9311 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:42.770638 9311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:42.770900 9311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:42.773310 9311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "9f5746ba32f6471caca47875a4cc386d"
format_stamp: "Formatted at 2026-09-29 03:14:42 on dist-test-slave-k1dg"
server_key: "317b68c941d207cf1ee902584a1e46d9"
server_key_iv: "bae85874ef395e6a2401d7463dd5d67a"
server_key_version: "encryptionkey@0"
I20260929 03:14:42.773988 9311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "9f5746ba32f6471caca47875a4cc386d"
format_stamp: "Formatted at 2026-09-29 03:14:42 on dist-test-slave-k1dg"
server_key: "317b68c941d207cf1ee902584a1e46d9"
server_key_iv: "bae85874ef395e6a2401d7463dd5d67a"
server_key_version: "encryptionkey@0"
I20260929 03:14:42.778357 9311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 03:14:42.781281 9326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:42.782838 9311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 03:14:42.783038 9311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "9f5746ba32f6471caca47875a4cc386d"
format_stamp: "Formatted at 2026-09-29 03:14:42 on dist-test-slave-k1dg"
server_key: "317b68c941d207cf1ee902584a1e46d9"
server_key_iv: "bae85874ef395e6a2401d7463dd5d67a"
server_key_version: "encryptionkey@0"
I20260929 03:14:42.783185 9311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:42.801178 9311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:42.804924 9311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:42.805145 9311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:42.813022 9311 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39199
I20260929 03:14:42.813051 9378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39199 every 8 connection(s)
I20260929 03:14:42.814101 9311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:42.816664 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 9311
I20260929 03:14:42.816830 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:42.817143 21177 external_mini_cluster.cc:1428] Setting key 1b5142e36bf82de534c3287260346cf3
I20260929 03:14:42.817590 9379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:42.824769 9379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d: Bootstrap starting.
I20260929 03:14:42.827512 9379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d: Neither blocks nor log segments found. Creating new log.
Sep 29 03:14:42 dist-test-slave-k1dg krb5kdc[9298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:14:42.828553 9379 log.cc:824] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:42.831061 9379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d: No bootstrap required, opened a new log
I20260929 03:14:42.834028 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:42.818812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58641 (local address 127.0.0.1:39199)
0929 03:14:42.819277 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:42.819294 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:42.819335 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:14:42.820103 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:42.820123 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:42.820497 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:42.821104 (+ 607us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:42.822048 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:42.823375 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:42.824398 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:42.825012 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:42.825194 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:42.828379 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:42.828386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:42.828402 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:42.828434 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:42.830746 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:42.831411 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:42.831416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:42.831421 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:42.831541 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:42.831881 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:42.831884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:42.831885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:42.832315 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:42.832468 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:42.832806 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 03:14:42.833164 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":160,"threads_started":1}
I20260929 03:14:42.835268 9379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5746ba32f6471caca47875a4cc386d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260929 03:14:42.835494 9379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:42.835559 9379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5746ba32f6471caca47875a4cc386d, State: Initialized, Role: FOLLOWER
I20260929 03:14:42.836198 9379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5746ba32f6471caca47875a4cc386d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260929 03:14:42.836426 9379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:42.836503 9379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:42.836652 9379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:42.837481 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:42.837882 9379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5746ba32f6471caca47875a4cc386d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260929 03:14:42.838256 9379 leader_election.cc:302] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f5746ba32f6471caca47875a4cc386d; no voters:
I20260929 03:14:42.838636 9379 leader_election.cc:288] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:42.838728 9384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:42.838991 9384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [term 1 LEADER]: Becoming Leader. State: Replica: 9f5746ba32f6471caca47875a4cc386d, State: Running, Role: LEADER
I20260929 03:14:42.839368 9384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5746ba32f6471caca47875a4cc386d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260929 03:14:42.840118 9379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:42.840816 9384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f5746ba32f6471caca47875a4cc386d. Latest consensus state: current_term: 1 leader_uuid: "9f5746ba32f6471caca47875a4cc386d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5746ba32f6471caca47875a4cc386d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } } }
I20260929 03:14:42.840948 9384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:42.841260 9387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f5746ba32f6471caca47875a4cc386d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5746ba32f6471caca47875a4cc386d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } } }
I20260929 03:14:42.841392 9387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d [sys.catalog]: This master's current role is: LEADER
W20260929 03:14:42.846465 9404 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:14:42.846571 9404 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:14:42.846632 9393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:42.847594 9393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:42.848979 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:42.839830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57723 (local address 127.0.0.1:39199)
0929 03:14:42.839964 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:42.839968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:42.840199 (+ 231us) server_negotiation.cc:407] Connection header received
0929 03:14:42.840402 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:42.840405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:42.840464 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:42.840553 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:42.841494 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:42.842080 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:42.843292 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:42.843647 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:42.843776 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:42.845495 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:42.845499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:42.845502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:42.845528 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:42.847181 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:42.847680 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:42.847684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:42.847686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:42.847787 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:42.848081 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:42.848084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:42.848086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:42.848297 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:42.848437 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:42.848680 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 03:14:42.848814 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":56}
I20260929 03:14:42.854336 9393 catalog_manager.cc:1347] Generated new cluster ID: d2d7a1fd32f34384b102a5def7fbc2ed
I20260929 03:14:42.854444 9393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:42.862005 9393 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:42.863458 9393 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:42.870885 9393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f5746ba32f6471caca47875a4cc386d: Generated new TSK 0
I20260929 03:14:42.871850 9393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:42.876343 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 9311
2026-09-29T03:14:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:42 dist-test-slave-k1dg krb5kdc[9410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:42 dist-test-slave-k1dg krb5kdc[9410](info): set up 2 sockets
Sep 29 03:14:42 dist-test-slave-k1dg krb5kdc[9410](info): commencing operation
krb5kdc: starting...
W20260929 03:14:43.962875 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:44 dist-test-slave-k1dg krb5kdc[9410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:44.122229 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:44.239491 9423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:44.239796 9423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:44.239840 9423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:44.244249 9423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:44.244338 9423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:44.244359 9423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:44.244375 9423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:44.244391 9423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:44.250135 9423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:44.251354 9423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:44.252511 9423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:44.261715 9428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:44.261832 9423 server_base.cc:1034] running on GCE node
W20260929 03:14:44.261844 9429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:44.262077 9431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:44.262733 9423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:44.264036 9423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:44.265296 9423 hybrid_clock.cc:648] HybridClock initialized: now 1790651684265257 us; error 58 us; skew 500 ppm
Sep 29 03:14:44 dist-test-slave-k1dg krb5kdc[9410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:44.268615 9423 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:44.269639 9423 webserver.cc:460] Webserver started at http://127.0.0.1:46107/ using document root <none> and password file <none>
I20260929 03:14:44.270254 9423 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:44.270318 9423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:44.270625 9423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:44.272950 9423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "783771a97af046de9cafca08ed0e0188"
format_stamp: "Formatted at 2026-09-29 03:14:44 on dist-test-slave-k1dg"
server_key: "1a90b3e0bac3ffc8dc7ca8426ca8ce59"
server_key_iv: "e624be607a5d8381ae39eeb831237bd3"
server_key_version: "encryptionkey@0"
I20260929 03:14:44.273614 9423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "783771a97af046de9cafca08ed0e0188"
format_stamp: "Formatted at 2026-09-29 03:14:44 on dist-test-slave-k1dg"
server_key: "1a90b3e0bac3ffc8dc7ca8426ca8ce59"
server_key_iv: "e624be607a5d8381ae39eeb831237bd3"
server_key_version: "encryptionkey@0"
I20260929 03:14:44.277897 9423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:14:44.281082 9438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:44.282846 9423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 03:14:44.283043 9423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "783771a97af046de9cafca08ed0e0188"
format_stamp: "Formatted at 2026-09-29 03:14:44 on dist-test-slave-k1dg"
server_key: "1a90b3e0bac3ffc8dc7ca8426ca8ce59"
server_key_iv: "e624be607a5d8381ae39eeb831237bd3"
server_key_version: "encryptionkey@0"
I20260929 03:14:44.283192 9423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:44.304004 9423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:44.307528 9423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:44.307767 9423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:44.315544 9423 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39517
I20260929 03:14:44.315570 9490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39517 every 8 connection(s)
I20260929 03:14:44.316605 9423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:44.319365 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 9423
I20260929 03:14:44.319519 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:44.319640 9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:44.319846 21177 external_mini_cluster.cc:1428] Setting key 30ba99ca90e9d5e2f65682684682e473
I20260929 03:14:44.327610 9491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188: Bootstrap starting.
I20260929 03:14:44.330739 9491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:44.330752 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:44.321409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46667 (local address 127.0.0.1:39517)
0929 03:14:44.321876 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:44.321912 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:44.321958 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:14:44.322739 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:44.322766 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:44.323232 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:44.323644 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:44.324080 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:44.325623 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:44.326692 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:44.327391 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:44.327557 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:44.328542 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:44.328548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:44.328564 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:44.329099 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:44.329176 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:44.329210 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:44.329312 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:44.329459 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:14:44.329684 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":127,"threads_started":1}
I20260929 03:14:44.331588 9491 log.cc:824] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:44.334033 9491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188: No bootstrap required, opened a new log
I20260929 03:14:44.334161 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:44.338452 9491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783771a97af046de9cafca08ed0e0188" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39517 } }
I20260929 03:14:44.338778 9491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:44.338840 9491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783771a97af046de9cafca08ed0e0188, State: Initialized, Role: FOLLOWER
I20260929 03:14:44.339284 9491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783771a97af046de9cafca08ed0e0188" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39517 } }
I20260929 03:14:44.339454 9491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:44.339546 9491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:44.339651 9491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:44.340821 9491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783771a97af046de9cafca08ed0e0188" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39517 } }
I20260929 03:14:44.341235 9491 leader_election.cc:302] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 783771a97af046de9cafca08ed0e0188; no voters:
I20260929 03:14:44.341620 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:44.336588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38533 (local address 127.0.0.1:39517)
0929 03:14:44.336746 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:44.336752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:44.337030 (+ 278us) server_negotiation.cc:407] Connection header received
0929 03:14:44.337171 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:44.337174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:44.337225 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:44.337314 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:44.337581 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:44.338318 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:44.339622 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:44.339971 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:44.340042 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:44.341069 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:44.341073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:44.341075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:44.341096 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:44.341122 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:44.341132 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:44.341270 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:44.341397 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:14:44.341454 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:14:44.341634 9491 leader_election.cc:288] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:44.341826 9501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:44.342058 9501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [term 1 LEADER]: Becoming Leader. State: Replica: 783771a97af046de9cafca08ed0e0188, State: Running, Role: LEADER
I20260929 03:14:44.342387 9501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783771a97af046de9cafca08ed0e0188" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39517 } }
I20260929 03:14:44.343211 9491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:44.343856 9502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "783771a97af046de9cafca08ed0e0188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783771a97af046de9cafca08ed0e0188" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39517 } } }
I20260929 03:14:44.344094 9502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:44.344383 9503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 783771a97af046de9cafca08ed0e0188. Latest consensus state: current_term: 1 leader_uuid: "783771a97af046de9cafca08ed0e0188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783771a97af046de9cafca08ed0e0188" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39517 } } }
I20260929 03:14:44.344482 9503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:44.344568 9510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:44.348115 9510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:44.354211 9510 catalog_manager.cc:1347] Generated new cluster ID: 5ac985ff0c7f499188b940099b67be22
I20260929 03:14:44.354295 9510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:44.361624 9510 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:44.362612 9510 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:44.380656 9510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 783771a97af046de9cafca08ed0e0188: Generated new TSK 0
I20260929 03:14:44.381686 9510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:44.389356 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 9423
2026-09-29T03:14:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:44 dist-test-slave-k1dg krb5kdc[9522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:44 dist-test-slave-k1dg krb5kdc[9522](info): set up 2 sockets
Sep 29 03:14:44 dist-test-slave-k1dg krb5kdc[9522](info): commencing operation
krb5kdc: starting...
W20260929 03:14:45.443470 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:45 dist-test-slave-k1dg krb5kdc[9522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:45.582342 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:45.706069 9535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:45.706388 9535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:45.706512 9535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:45.711278 9535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:45.711366 9535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:45.711390 9535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:45.711407 9535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:45.711423 9535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:45.717219 9535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:45.718396 9535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:45.719537 9535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:45.729151 9541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:45.729279 9535 server_base.cc:1034] running on GCE node
W20260929 03:14:45.729167 9543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:45.729167 9540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:45.730042 9535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:45.731438 9535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:45.732625 9535 hybrid_clock.cc:648] HybridClock initialized: now 1790651685732588 us; error 63 us; skew 500 ppm
Sep 29 03:14:45 dist-test-slave-k1dg krb5kdc[9522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:45.735867 9535 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:45.736987 9535 webserver.cc:460] Webserver started at http://127.0.0.1:46821/ using document root <none> and password file <none>
I20260929 03:14:45.737634 9535 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:45.737739 9535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:45.737993 9535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:45.740252 9535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "ae89fe78e1a449e39c9a0ae166b74db7"
format_stamp: "Formatted at 2026-09-29 03:14:45 on dist-test-slave-k1dg"
server_key: "c1c15941c1b0fd2221ad0b48e2e00e84"
server_key_iv: "313a4e917d4024abeeb7fda621105e96"
server_key_version: "encryptionkey@0"
I20260929 03:14:45.740856 9535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "ae89fe78e1a449e39c9a0ae166b74db7"
format_stamp: "Formatted at 2026-09-29 03:14:45 on dist-test-slave-k1dg"
server_key: "c1c15941c1b0fd2221ad0b48e2e00e84"
server_key_iv: "313a4e917d4024abeeb7fda621105e96"
server_key_version: "encryptionkey@0"
I20260929 03:14:45.744972 9535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 03:14:45.747722 9550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:45.749186 9535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:14:45.749356 9535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "ae89fe78e1a449e39c9a0ae166b74db7"
format_stamp: "Formatted at 2026-09-29 03:14:45 on dist-test-slave-k1dg"
server_key: "c1c15941c1b0fd2221ad0b48e2e00e84"
server_key_iv: "313a4e917d4024abeeb7fda621105e96"
server_key_version: "encryptionkey@0"
I20260929 03:14:45.749521 9535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:45.754496 9535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:45.757733 9535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:45.757957 9535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:45.765604 9535 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40681
I20260929 03:14:45.765627 9602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40681 every 8 connection(s)
I20260929 03:14:45.766674 9535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:45.769368 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 9535
I20260929 03:14:45.769528 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:45.769856 21177 external_mini_cluster.cc:1428] Setting key ebeb736beb9ad7080b872162c8ca24ae
I20260929 03:14:45.769878 9603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:45.775645 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:45.771421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59397 (local address 127.0.0.1:40681)
0929 03:14:45.771922 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:45.771938 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:45.771974 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:14:45.772862 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:45.772907 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:45.773208 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:45.773574 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:45.774371 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:45.774377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:45.774391 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:45.774401 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:45.774513 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:45.774537 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:45.774691 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:45.774765 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 03:14:45.774941 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":148,"threads_started":1}
I20260929 03:14:45.777180 9603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7: Bootstrap starting.
I20260929 03:14:45.778790 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:45.779886 9603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:45.780707 9603 log.cc:824] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:45.782789 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:45.780875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45201 (local address 127.0.0.1:40681)
0929 03:14:45.781007 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:45.781012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:45.781243 (+ 231us) server_negotiation.cc:407] Connection header received
0929 03:14:45.781406 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:45.781409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:45.781452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:45.781536 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:45.782278 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:45.782281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:45.782284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:45.782295 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:45.782330 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:45.782341 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:45.782449 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:45.782621 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 03:14:45.782661 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:14:45.783157 9603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7: No bootstrap required, opened a new log
I20260929 03:14:45.786927 9603 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89fe78e1a449e39c9a0ae166b74db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } }
I20260929 03:14:45.787200 9603 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:45.787245 9603 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae89fe78e1a449e39c9a0ae166b74db7, State: Initialized, Role: FOLLOWER
I20260929 03:14:45.787667 9603 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89fe78e1a449e39c9a0ae166b74db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } }
I20260929 03:14:45.787784 9603 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:45.787835 9603 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:45.787909 9603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:45.788870 9603 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89fe78e1a449e39c9a0ae166b74db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } }
I20260929 03:14:45.789188 9603 leader_election.cc:302] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae89fe78e1a449e39c9a0ae166b74db7; no voters:
I20260929 03:14:45.789582 9603 leader_election.cc:288] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:45.789716 9613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:45.789963 9613 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [term 1 LEADER]: Becoming Leader. State: Replica: ae89fe78e1a449e39c9a0ae166b74db7, State: Running, Role: LEADER
I20260929 03:14:45.790366 9613 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89fe78e1a449e39c9a0ae166b74db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } }
I20260929 03:14:45.791100 9603 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:45.792295 9614 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae89fe78e1a449e39c9a0ae166b74db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89fe78e1a449e39c9a0ae166b74db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } } }
I20260929 03:14:45.792452 9614 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:45.792871 9622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:45.792812 9615 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae89fe78e1a449e39c9a0ae166b74db7. Latest consensus state: current_term: 1 leader_uuid: "ae89fe78e1a449e39c9a0ae166b74db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89fe78e1a449e39c9a0ae166b74db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40681 } } }
I20260929 03:14:45.793023 9615 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:45.796197 9622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:45.802577 9622 catalog_manager.cc:1347] Generated new cluster ID: cf379e0ba1c94edda27056af79ec8bf0
I20260929 03:14:45.802668 9622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:45.817130 9622 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:45.817549 9622 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:45.829974 9622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae89fe78e1a449e39c9a0ae166b74db7: Generated new TSK 0
I20260929 03:14:45.830873 9622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:45.842489 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 9535
2026-09-29T03:14:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:45 dist-test-slave-k1dg krb5kdc[9634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:45 dist-test-slave-k1dg krb5kdc[9634](info): set up 2 sockets
Sep 29 03:14:45 dist-test-slave-k1dg krb5kdc[9634](info): commencing operation
krb5kdc: starting...
W20260929 03:14:46.914969 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:47 dist-test-slave-k1dg krb5kdc[9634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:47.050915 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:47.167309 9647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:47.167667 9647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:47.167778 9647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:47.172060 9647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:47.172166 9647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:47.172207 9647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:47.172264 9647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:47.172305 9647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:47.177815 9647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:47.179179 9647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:47.180335 9647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:47.189827 9652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:47.189841 9653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:47.189870 9655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:47.190143 9647 server_base.cc:1034] running on GCE node
I20260929 03:14:47.190984 9647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:47.192292 9647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:47.193516 9647 hybrid_clock.cc:648] HybridClock initialized: now 1790651687193498 us; error 34 us; skew 500 ppm
Sep 29 03:14:47 dist-test-slave-k1dg krb5kdc[9634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:47.196529 9647 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:47.197525 9647 webserver.cc:460] Webserver started at http://127.0.0.1:37221/ using document root <none> and password file <none>
I20260929 03:14:47.198073 9647 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:47.198135 9647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:47.198313 9647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:47.200556 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "3069f2f3db70446b9db1d6913e27fdf2"
format_stamp: "Formatted at 2026-09-29 03:14:47 on dist-test-slave-k1dg"
server_key: "3e4e1d7936da12c9a38a9e7535c4ead6"
server_key_iv: "208d92d1612aafbd8381214b79a59f62"
server_key_version: "encryptionkey@0"
I20260929 03:14:47.201102 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "3069f2f3db70446b9db1d6913e27fdf2"
format_stamp: "Formatted at 2026-09-29 03:14:47 on dist-test-slave-k1dg"
server_key: "3e4e1d7936da12c9a38a9e7535c4ead6"
server_key_iv: "208d92d1612aafbd8381214b79a59f62"
server_key_version: "encryptionkey@0"
I20260929 03:14:47.205276 9647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 03:14:47.208154 9662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:47.209751 9647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:14:47.209885 9647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "3069f2f3db70446b9db1d6913e27fdf2"
format_stamp: "Formatted at 2026-09-29 03:14:47 on dist-test-slave-k1dg"
server_key: "3e4e1d7936da12c9a38a9e7535c4ead6"
server_key_iv: "208d92d1612aafbd8381214b79a59f62"
server_key_version: "encryptionkey@0"
I20260929 03:14:47.210000 9647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:47.222198 9647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:47.225772 9647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:47.225955 9647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:47.233909 9647 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41231
I20260929 03:14:47.233928 9714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41231 every 8 connection(s)
I20260929 03:14:47.235059 9647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:47.237784 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 9647
I20260929 03:14:47.237960 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:47.238274 21177 external_mini_cluster.cc:1428] Setting key 146437531cf038e389a0b45f1feec0fc
I20260929 03:14:47.238399 9715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:47.246263 9715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2: Bootstrap starting.
Sep 29 03:14:47 dist-test-slave-k1dg krb5kdc[9634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:14:47.249684 9715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:47.250631 9715 log.cc:824] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:47.252888 9715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2: No bootstrap required, opened a new log
I20260929 03:14:47.255538 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:47.240005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:56429 (local address 127.0.0.1:41231)
0929 03:14:47.240668 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:47.240690 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:47.240733 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:14:47.241582 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:47.241620 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:47.242056 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:47.242813 (+ 757us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:47.243820 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:47.244902 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:47.245883 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:47.246409 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:47.246635 (+ 226us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:47.250106 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:47.250112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:47.250128 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:47.250163 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:47.252149 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:47.252763 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:47.252769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:47.252774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:47.252904 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:47.253315 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:47.253321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:47.253327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:47.253866 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:47.254115 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:47.254502 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 03:14:47.254739 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":205,"threads_started":1}
I20260929 03:14:47.257452 9715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3069f2f3db70446b9db1d6913e27fdf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } }
I20260929 03:14:47.257754 9715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:47.257856 9715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3069f2f3db70446b9db1d6913e27fdf2, State: Initialized, Role: FOLLOWER
I20260929 03:14:47.258416 9715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3069f2f3db70446b9db1d6913e27fdf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } }
I20260929 03:14:47.258637 9715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:47.258711 9715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:47.258813 9715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:47.259177 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:47.259928 9715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3069f2f3db70446b9db1d6913e27fdf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } }
I20260929 03:14:47.260301 9715 leader_election.cc:302] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3069f2f3db70446b9db1d6913e27fdf2; no voters:
I20260929 03:14:47.260673 9715 leader_election.cc:288] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:47.260747 9723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:47.260967 9723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [term 1 LEADER]: Becoming Leader. State: Replica: 3069f2f3db70446b9db1d6913e27fdf2, State: Running, Role: LEADER
I20260929 03:14:47.261328 9723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3069f2f3db70446b9db1d6913e27fdf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } }
I20260929 03:14:47.262125 9715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:47.262686 9725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3069f2f3db70446b9db1d6913e27fdf2. Latest consensus state: current_term: 1 leader_uuid: "3069f2f3db70446b9db1d6913e27fdf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3069f2f3db70446b9db1d6913e27fdf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } } }
I20260929 03:14:47.262827 9725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:47.263419 9726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3069f2f3db70446b9db1d6913e27fdf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3069f2f3db70446b9db1d6913e27fdf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } } }
I20260929 03:14:47.263617 9726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2 [sys.catalog]: This master's current role is: LEADER
W20260929 03:14:47.267405 9740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:14:47.267516 9740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:14:47.267894 9734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:47.268692 9734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:47.273403 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:47.261469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:53181 (local address 127.0.0.1:41231)
0929 03:14:47.261589 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:47.261593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:47.262114 (+ 521us) server_negotiation.cc:407] Connection header received
0929 03:14:47.262316 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:47.262319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:47.262375 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:47.262498 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:47.264830 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:47.265489 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:47.266728 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:47.267096 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:47.267222 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:47.269034 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:47.269039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:47.269042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:47.269071 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:47.271045 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:47.271816 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:47.271819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:47.271821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:47.271927 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:47.272346 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:47.272349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:47.272351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:47.272578 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:47.272725 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:47.273086 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 03:14:47.273206 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:14:47.275883 9734 catalog_manager.cc:1347] Generated new cluster ID: 96c17338f6c84da39c6914085b2e0a71
I20260929 03:14:47.276000 9734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:47.294808 9734 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:47.295796 9734 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:47.301003 9734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3069f2f3db70446b9db1d6913e27fdf2: Generated new TSK 0
I20260929 03:14:47.301905 9734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:47.314599 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 9647
W20260929 03:14:47.320155 9632 connection.cc:535] client connection to 127.0.0.1:41231 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41231): Connection reset by peer (error 104)
2026-09-29T03:14:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:47 dist-test-slave-k1dg krb5kdc[9746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:47 dist-test-slave-k1dg krb5kdc[9746](info): set up 2 sockets
Sep 29 03:14:47 dist-test-slave-k1dg krb5kdc[9746](info): commencing operation
krb5kdc: starting...
W20260929 03:14:48.406785 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:48 dist-test-slave-k1dg krb5kdc[9746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:48.537978 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:48.658598 9759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:48.658905 9759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:48.658949 9759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:48.663259 9759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:48.663339 9759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:48.663359 9759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:48.663378 9759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:48.663429 9759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:48.669433 9759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:48.670645 9759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:48.671757 9759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:48.681265 9764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:48.681213 9767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:48.681291 9759 server_base.cc:1034] running on GCE node
W20260929 03:14:48.681237 9765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:48.682284 9759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:48.683573 9759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:48.684813 9759 hybrid_clock.cc:648] HybridClock initialized: now 1790651688684792 us; error 45 us; skew 500 ppm
Sep 29 03:14:48 dist-test-slave-k1dg krb5kdc[9746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:48.688485 9759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:48.689718 9759 webserver.cc:460] Webserver started at http://127.0.0.1:38053/ using document root <none> and password file <none>
I20260929 03:14:48.690382 9759 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:48.690526 9759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:48.690809 9759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:48.693159 9759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "f5df949fe01b4ae6b9ed153e8f9d0ecf"
format_stamp: "Formatted at 2026-09-29 03:14:48 on dist-test-slave-k1dg"
server_key: "de7a1113860910228c15f49216fddf3a"
server_key_iv: "0e8d2f3d729da5afd4c804a0b6a3a38a"
server_key_version: "encryptionkey@0"
I20260929 03:14:48.693857 9759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "f5df949fe01b4ae6b9ed153e8f9d0ecf"
format_stamp: "Formatted at 2026-09-29 03:14:48 on dist-test-slave-k1dg"
server_key: "de7a1113860910228c15f49216fddf3a"
server_key_iv: "0e8d2f3d729da5afd4c804a0b6a3a38a"
server_key_version: "encryptionkey@0"
I20260929 03:14:48.698799 9759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260929 03:14:48.701813 9774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:48.703421 9759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:48.703598 9759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "f5df949fe01b4ae6b9ed153e8f9d0ecf"
format_stamp: "Formatted at 2026-09-29 03:14:48 on dist-test-slave-k1dg"
server_key: "de7a1113860910228c15f49216fddf3a"
server_key_iv: "0e8d2f3d729da5afd4c804a0b6a3a38a"
server_key_version: "encryptionkey@0"
I20260929 03:14:48.703770 9759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:48.725107 9759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:48.729139 9759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:48.729400 9759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:48.737715 9759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39505
I20260929 03:14:48.737794 9826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39505 every 8 connection(s)
I20260929 03:14:48.738878 9759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:48.741914 9827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:48.745749 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 9759
I20260929 03:14:48.745923 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:48.746224 21177 external_mini_cluster.cc:1428] Setting key f4503b39ac233a08a63fdeb83cd7f510
I20260929 03:14:48.748356 9827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf: Bootstrap starting.
I20260929 03:14:48.751012 9827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:48.751950 9827 log.cc:824] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:48.754488 9827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf: No bootstrap required, opened a new log
I20260929 03:14:48.756803 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:48.747881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:43875 (local address 127.0.0.1:39505)
0929 03:14:48.748353 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:48.748374 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:48.748438 (+ 64us) server_negotiation.cc:407] Connection header received
0929 03:14:48.749232 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:48.749252 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:48.749632 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:48.749945 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:48.750288 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:48.751455 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:48.752724 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:48.753255 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:48.753457 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:48.754601 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:48.754610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:48.754629 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:48.755077 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:48.755196 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:48.755225 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:48.755383 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:48.755598 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 03:14:48.755757 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":173,"threads_started":1}
I20260929 03:14:48.757915 9827 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df949fe01b4ae6b9ed153e8f9d0ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } }
I20260929 03:14:48.758215 9827 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:48.758268 9827 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5df949fe01b4ae6b9ed153e8f9d0ecf, State: Initialized, Role: FOLLOWER
I20260929 03:14:48.758795 9827 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df949fe01b4ae6b9ed153e8f9d0ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } }
I20260929 03:14:48.758983 9827 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:48.759057 9827 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:48.759220 9827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:48.760293 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:48.760514 9827 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df949fe01b4ae6b9ed153e8f9d0ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } }
I20260929 03:14:48.760914 9827 leader_election.cc:302] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5df949fe01b4ae6b9ed153e8f9d0ecf; no voters:
I20260929 03:14:48.761300 9827 leader_election.cc:288] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:48.761485 9832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:48.761921 9832 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [term 1 LEADER]: Becoming Leader. State: Replica: f5df949fe01b4ae6b9ed153e8f9d0ecf, State: Running, Role: LEADER
I20260929 03:14:48.762319 9832 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df949fe01b4ae6b9ed153e8f9d0ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } }
I20260929 03:14:48.762943 9827 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:48.763906 9832 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5df949fe01b4ae6b9ed153e8f9d0ecf. Latest consensus state: current_term: 1 leader_uuid: "f5df949fe01b4ae6b9ed153e8f9d0ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df949fe01b4ae6b9ed153e8f9d0ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } } }
I20260929 03:14:48.764110 9832 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:48.764357 9833 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5df949fe01b4ae6b9ed153e8f9d0ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df949fe01b4ae6b9ed153e8f9d0ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39505 } } }
I20260929 03:14:48.764470 9833 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:48.764627 9846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:48.768435 9846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:48.770474 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:48.762618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:54977 (local address 127.0.0.1:39505)
0929 03:14:48.762733 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:48.762738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:48.765728 (+ 2990us) server_negotiation.cc:407] Connection header received
0929 03:14:48.765928 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:48.765932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:48.765988 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:48.766097 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:48.766409 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:48.766971 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:48.768262 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:48.768666 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:48.768776 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:48.769920 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:48.769924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:48.769927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:48.769943 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:48.769983 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:48.769994 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:48.770100 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:48.770239 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 03:14:48.770276 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:14:48.775521 9846 catalog_manager.cc:1347] Generated new cluster ID: dcdc8a5e33954fab99158268cc66f8f0
I20260929 03:14:48.775601 9846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:48.800055 9846 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:48.801841 9846 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:48.807462 9846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5df949fe01b4ae6b9ed153e8f9d0ecf: Generated new TSK 0
I20260929 03:14:48.808425 9846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:48.820299 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 9759
W20260929 03:14:48.825958 9744 connection.cc:535] client connection to 127.0.0.1:39505 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39505): Connection reset by peer (error 104)
2026-09-29T03:14:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1505 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:48 dist-test-slave-k1dg krb5kdc[9858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:48 dist-test-slave-k1dg krb5kdc[9858](info): set up 2 sockets
Sep 29 03:14:48 dist-test-slave-k1dg krb5kdc[9858](info): commencing operation
krb5kdc: starting...
W20260929 03:14:49.879077 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:49 dist-test-slave-k1dg krb5kdc[9858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:50.014406 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:50.131831 9871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:50.132217 9871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:50.132316 9871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:50.136662 9871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:50.136783 9871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:50.136824 9871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:50.136873 9871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:50.136919 9871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:50.142732 9871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:50.143966 9871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:50.145066 9871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:14:50.154542 9871 server_base.cc:1034] running on GCE node
W20260929 03:14:50.154595 9876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:50.154580 9879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:50.154773 9877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:50.155352 9871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:50.156538 9871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:50.157721 9871 hybrid_clock.cc:648] HybridClock initialized: now 1790651690157680 us; error 50 us; skew 500 ppm
Sep 29 03:14:50 dist-test-slave-k1dg krb5kdc[9858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:50.160843 9871 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:50.161761 9871 webserver.cc:460] Webserver started at http://127.0.0.1:46319/ using document root <none> and password file <none>
I20260929 03:14:50.162297 9871 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:50.162355 9871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:50.162603 9871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:50.164662 9871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "878fdc69a2274d4f887d42aa3cd999ef"
format_stamp: "Formatted at 2026-09-29 03:14:50 on dist-test-slave-k1dg"
server_key: "6bf3955d5d9037f2279c08927ed7802a"
server_key_iv: "f4629a3e07084421d8087f8b3472f6e9"
server_key_version: "encryptionkey@0"
I20260929 03:14:50.165191 9871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "878fdc69a2274d4f887d42aa3cd999ef"
format_stamp: "Formatted at 2026-09-29 03:14:50 on dist-test-slave-k1dg"
server_key: "6bf3955d5d9037f2279c08927ed7802a"
server_key_iv: "f4629a3e07084421d8087f8b3472f6e9"
server_key_version: "encryptionkey@0"
I20260929 03:14:50.169044 9871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 03:14:50.171537 9886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:50.173005 9871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:50.173120 9871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "878fdc69a2274d4f887d42aa3cd999ef"
format_stamp: "Formatted at 2026-09-29 03:14:50 on dist-test-slave-k1dg"
server_key: "6bf3955d5d9037f2279c08927ed7802a"
server_key_iv: "f4629a3e07084421d8087f8b3472f6e9"
server_key_version: "encryptionkey@0"
I20260929 03:14:50.173223 9871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:50.177947 9871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:50.181573 9871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:50.181780 9871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:50.188925 9871 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33999
I20260929 03:14:50.188951 9938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33999 every 8 connection(s)
I20260929 03:14:50.189941 9871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:50.191267 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 9871
I20260929 03:14:50.191416 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:50.191720 21177 external_mini_cluster.cc:1428] Setting key 41d9bf7777ba1dd80db622b854fdaa00
I20260929 03:14:50.193289 9939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:50.197778 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:50.193324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:48659 (local address 127.0.0.1:33999)
0929 03:14:50.193925 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:50.193948 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:50.194012 (+ 64us) server_negotiation.cc:407] Connection header received
0929 03:14:50.194832 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:50.194853 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:50.195164 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:50.195535 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:50.196472 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:50.196477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:50.196490 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:50.196501 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:50.196559 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:50.196579 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:50.196746 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:50.196822 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 03:14:50.196997 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":153,"threads_started":1}
I20260929 03:14:50.200399 9939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef: Bootstrap starting.
I20260929 03:14:50.201125 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:50.203258 9939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:50.204416 9939 log.cc:824] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:50.205842 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:50.203546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:44425 (local address 127.0.0.1:33999)
0929 03:14:50.203672 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:50.203676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:50.204125 (+ 449us) server_negotiation.cc:407] Connection header received
0929 03:14:50.204332 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:50.204335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:50.204372 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:50.204457 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:50.205278 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:50.205281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:50.205283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:50.205293 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:50.205318 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:50.205329 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:50.205409 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:50.205584 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 03:14:50.205632 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:14:50.207123 9939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef: No bootstrap required, opened a new log
I20260929 03:14:50.210853 9939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878fdc69a2274d4f887d42aa3cd999ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } }
I20260929 03:14:50.211105 9939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:50.211141 9939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878fdc69a2274d4f887d42aa3cd999ef, State: Initialized, Role: FOLLOWER
I20260929 03:14:50.211648 9939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878fdc69a2274d4f887d42aa3cd999ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } }
I20260929 03:14:50.211833 9939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:50.211905 9939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:50.212038 9939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:50.213018 9939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878fdc69a2274d4f887d42aa3cd999ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } }
I20260929 03:14:50.213392 9939 leader_election.cc:302] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 878fdc69a2274d4f887d42aa3cd999ef; no voters:
I20260929 03:14:50.213732 9939 leader_election.cc:288] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:50.213887 9949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:50.214125 9949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [term 1 LEADER]: Becoming Leader. State: Replica: 878fdc69a2274d4f887d42aa3cd999ef, State: Running, Role: LEADER
I20260929 03:14:50.214557 9949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878fdc69a2274d4f887d42aa3cd999ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } }
I20260929 03:14:50.215129 9939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:50.216320 9951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 878fdc69a2274d4f887d42aa3cd999ef. Latest consensus state: current_term: 1 leader_uuid: "878fdc69a2274d4f887d42aa3cd999ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878fdc69a2274d4f887d42aa3cd999ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } } }
I20260929 03:14:50.216455 9951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:50.216668 9950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "878fdc69a2274d4f887d42aa3cd999ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878fdc69a2274d4f887d42aa3cd999ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } } }
I20260929 03:14:50.216822 9950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:50.216876 9958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:50.220435 9958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:50.227025 9958 catalog_manager.cc:1347] Generated new cluster ID: 5067a35c4d9047ccbf2bc41a4ac5c942
I20260929 03:14:50.227119 9958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:50.250264 9958 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:50.250865 9958 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:50.258034 9958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 878fdc69a2274d4f887d42aa3cd999ef: Generated new TSK 0
I20260929 03:14:50.258917 9958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:50.264927 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 9871
2026-09-29T03:14:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:50 dist-test-slave-k1dg krb5kdc[9970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:50 dist-test-slave-k1dg krb5kdc[9970](info): set up 2 sockets
Sep 29 03:14:50 dist-test-slave-k1dg krb5kdc[9970](info): commencing operation
krb5kdc: starting...
W20260929 03:14:51.330994 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:51 dist-test-slave-k1dg krb5kdc[9970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:51.462416 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:51.580466 9983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:51.580840 9983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:51.580930 9983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:51.585309 9983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:51.585440 9983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:51.585480 9983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:51.585520 9983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:51.585554 9983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:51.591791 9983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45969
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:51.593070 9983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:51.594735 9983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:51.603916 9988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:51.603902 9989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:51.604115 9983 server_base.cc:1034] running on GCE node
W20260929 03:14:51.603909 9991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:51.604799 9983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:51.606127 9983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:51.607340 9983 hybrid_clock.cc:648] HybridClock initialized: now 1790651691607322 us; error 34 us; skew 500 ppm
Sep 29 03:14:51 dist-test-slave-k1dg krb5kdc[9970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:51.610409 9983 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:51.611450 9983 webserver.cc:460] Webserver started at http://127.0.0.1:41239/ using document root <none> and password file <none>
I20260929 03:14:51.612006 9983 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:51.612068 9983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:51.612243 9983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:51.614355 9983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "42bd187412f944fbaec57d902a76b0bd"
format_stamp: "Formatted at 2026-09-29 03:14:51 on dist-test-slave-k1dg"
server_key: "2a38f5f8277d9fcfde9cb1775c5481dc"
server_key_iv: "029667a7727e01123f8a0721278ba75e"
server_key_version: "encryptionkey@0"
I20260929 03:14:51.615015 9983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "42bd187412f944fbaec57d902a76b0bd"
format_stamp: "Formatted at 2026-09-29 03:14:51 on dist-test-slave-k1dg"
server_key: "2a38f5f8277d9fcfde9cb1775c5481dc"
server_key_iv: "029667a7727e01123f8a0721278ba75e"
server_key_version: "encryptionkey@0"
I20260929 03:14:51.618970 9983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:14:51.621596 9998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:51.623047 9983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:51.623186 9983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "42bd187412f944fbaec57d902a76b0bd"
format_stamp: "Formatted at 2026-09-29 03:14:51 on dist-test-slave-k1dg"
server_key: "2a38f5f8277d9fcfde9cb1775c5481dc"
server_key_iv: "029667a7727e01123f8a0721278ba75e"
server_key_version: "encryptionkey@0"
I20260929 03:14:51.623306 9983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:51.646744 9983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:51.650682 9983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:51.650913 9983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:51.658949 9983 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45969
I20260929 03:14:51.659197 10050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45969 every 8 connection(s)
I20260929 03:14:51.660111 9983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:51.663239 10051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:51.669250 10051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd: Bootstrap starting.
I20260929 03:14:51.670233 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 9983
I20260929 03:14:51.670408 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:51.670758 21177 external_mini_cluster.cc:1428] Setting key 0012dfd20d57b5e5f4b69b5d767eabf6
I20260929 03:14:51.672613 10051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:51.673499 10051 log.cc:824] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:51.675967 10051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd: No bootstrap required, opened a new log
I20260929 03:14:51.681365 10051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bd187412f944fbaec57d902a76b0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } }
I20260929 03:14:51.681658 10051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:51.681761 10051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42bd187412f944fbaec57d902a76b0bd, State: Initialized, Role: FOLLOWER
Sep 29 03:14:51 dist-test-slave-k1dg krb5kdc[9970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:14:51.682374 10051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bd187412f944fbaec57d902a76b0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } }
I20260929 03:14:51.682675 10051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:51.682773 10051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:51.682888 10051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:51.684324 10051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bd187412f944fbaec57d902a76b0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } }
I20260929 03:14:51.684753 10051 leader_election.cc:302] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42bd187412f944fbaec57d902a76b0bd; no voters:
I20260929 03:14:51.685160 10051 leader_election.cc:288] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:51.685277 10056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:51.685572 10056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [term 1 LEADER]: Becoming Leader. State: Replica: 42bd187412f944fbaec57d902a76b0bd, State: Running, Role: LEADER
I20260929 03:14:51.686050 10056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bd187412f944fbaec57d902a76b0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } }
I20260929 03:14:51.686848 10051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:51.687615 10056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42bd187412f944fbaec57d902a76b0bd. Latest consensus state: current_term: 1 leader_uuid: "42bd187412f944fbaec57d902a76b0bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bd187412f944fbaec57d902a76b0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } } }
I20260929 03:14:51.687844 10056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:51.688107 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42bd187412f944fbaec57d902a76b0bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bd187412f944fbaec57d902a76b0bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45969 } } }
I20260929 03:14:51.688220 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:51.689829 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:51.672003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:41413 (local address 127.0.0.1:45969)
0929 03:14:51.672500 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:51.672517 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:51.672557 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:14:51.673453 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:51.673477 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:51.673996 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:51.674414 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:51.675399 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:51.676475 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:51.677739 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:51.678765 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:51.678975 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:51.682714 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:51.682723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:51.682745 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:51.682788 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:51.685202 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:51.685935 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:51.685941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:51.685948 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:51.686113 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:51.686611 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:51.686614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:51.686616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:51.687163 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:51.688224 (+ 1061us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:51.688418 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 03:14:51.688704 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":143,"threads_started":1}
W20260929 03:14:51.692282 10071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:14:51.692440 10071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:14:51.692898 10065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:51.693650 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:51.693954 10065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:51.700529 10065 catalog_manager.cc:1347] Generated new cluster ID: ab46e89ff3a147e1bc83d4f2a262ad53
I20260929 03:14:51.700712 10065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:51.705377 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:51.696086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:52311 (local address 127.0.0.1:45969)
0929 03:14:51.696217 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:51.696222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:51.696539 (+ 317us) server_negotiation.cc:407] Connection header received
0929 03:14:51.696648 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:51.696651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:51.696705 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:51.696811 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:51.697691 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:51.698615 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:51.699530 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:51.699786 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:51.699902 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:51.701673 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:51.701676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:51.701681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:51.701712 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:51.703454 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:51.703999 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:51.704003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:51.704005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:51.704081 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:51.704416 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:51.704419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:51.704420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:51.704609 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:51.704730 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:51.705119 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 03:14:51.705217 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:14:51.722808 10065 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:51.724182 10065 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:51.732183 10065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42bd187412f944fbaec57d902a76b0bd: Generated new TSK 0
I20260929 03:14:51.733112 10065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:51.741573 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 9983
W20260929 03:14:51.747210 9968 connection.cc:535] client connection to 127.0.0.1:45969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45969): Connection reset by peer (error 104)
2026-09-29T03:14:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:51 dist-test-slave-k1dg krb5kdc[10082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:51 dist-test-slave-k1dg krb5kdc[10082](info): set up 2 sockets
Sep 29 03:14:51 dist-test-slave-k1dg krb5kdc[10082](info): commencing operation
krb5kdc: starting...
W20260929 03:14:52.834894 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:52 dist-test-slave-k1dg krb5kdc[10082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:52.964294 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:53.076614 10095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:53.076967 10095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:53.077045 10095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:53.081209 10095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:53.081336 10095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:53.081375 10095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:53.081429 10095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:53.081465 10095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:53.087157 10095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:53.088359 10095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:53.089424 10095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:53.098040 10103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:53.098258 10095 server_base.cc:1034] running on GCE node
W20260929 03:14:53.098184 10101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:53.098040 10100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:53.098932 10095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:53.100062 10095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:53.101235 10095 hybrid_clock.cc:648] HybridClock initialized: now 1790651693101196 us; error 54 us; skew 500 ppm
Sep 29 03:14:53 dist-test-slave-k1dg krb5kdc[10082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:53.104168 10095 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:53.105216 10095 webserver.cc:460] Webserver started at http://127.0.0.1:39137/ using document root <none> and password file <none>
I20260929 03:14:53.105846 10095 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:53.105911 10095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:53.106125 10095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:53.108284 10095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "dbb06f30c6854c7ebedc240126092f22"
format_stamp: "Formatted at 2026-09-29 03:14:53 on dist-test-slave-k1dg"
server_key: "35d79050fed8b5783a0eacf2c1398d38"
server_key_iv: "54fa87eff0ad5a1517ba38c57bcf2db9"
server_key_version: "encryptionkey@0"
I20260929 03:14:53.108860 10095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "dbb06f30c6854c7ebedc240126092f22"
format_stamp: "Formatted at 2026-09-29 03:14:53 on dist-test-slave-k1dg"
server_key: "35d79050fed8b5783a0eacf2c1398d38"
server_key_iv: "54fa87eff0ad5a1517ba38c57bcf2db9"
server_key_version: "encryptionkey@0"
I20260929 03:14:53.112797 10095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:14:53.115334 10110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:53.116640 10095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:14:53.116824 10095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "dbb06f30c6854c7ebedc240126092f22"
format_stamp: "Formatted at 2026-09-29 03:14:53 on dist-test-slave-k1dg"
server_key: "35d79050fed8b5783a0eacf2c1398d38"
server_key_iv: "54fa87eff0ad5a1517ba38c57bcf2db9"
server_key_version: "encryptionkey@0"
I20260929 03:14:53.116941 10095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:53.135442 10095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:53.138955 10095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:53.139191 10095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:53.146512 10095 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41943
I20260929 03:14:53.146536 10162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41943 every 8 connection(s)
I20260929 03:14:53.147666 10095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:53.150583 10163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:53.151539 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 10095
I20260929 03:14:53.151686 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:53.151996 21177 external_mini_cluster.cc:1428] Setting key 1ffdba7ad4f29f52102486d8eb13a712
I20260929 03:14:53.158397 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22: Bootstrap starting.
I20260929 03:14:53.161108 10163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:53.161855 10163 log.cc:824] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:53.161897 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:53.153314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:55005 (local address 127.0.0.1:41943)
0929 03:14:53.153739 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:53.153755 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:53.153792 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:14:53.154607 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:53.154627 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:53.154983 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:53.155278 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:53.155649 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:53.156620 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:53.157965 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:53.158686 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:53.158850 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:53.160049 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:53.160054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:53.160067 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:53.160481 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:53.160538 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:53.160563 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:53.160665 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:53.160948 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 03:14:53.161096 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":145,"threads_started":1}
I20260929 03:14:53.164112 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22: No bootstrap required, opened a new log
I20260929 03:14:53.165076 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:53.168224 10163 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb06f30c6854c7ebedc240126092f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260929 03:14:53.168458 10163 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:53.168521 10163 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb06f30c6854c7ebedc240126092f22, State: Initialized, Role: FOLLOWER
I20260929 03:14:53.169014 10163 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb06f30c6854c7ebedc240126092f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260929 03:14:53.169178 10163 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:53.169266 10163 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:53.169363 10163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:53.170300 10163 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb06f30c6854c7ebedc240126092f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260929 03:14:53.170687 10163 leader_election.cc:302] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbb06f30c6854c7ebedc240126092f22; no voters:
I20260929 03:14:53.171007 10163 leader_election.cc:288] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:53.171118 10173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:53.171350 10173 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [term 1 LEADER]: Becoming Leader. State: Replica: dbb06f30c6854c7ebedc240126092f22, State: Running, Role: LEADER
I20260929 03:14:53.171769 10173 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb06f30c6854c7ebedc240126092f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260929 03:14:53.172394 10163 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:53.173116 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:53.167109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:53805 (local address 127.0.0.1:41943)
0929 03:14:53.167223 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:53.167226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:53.168828 (+ 1602us) server_negotiation.cc:407] Connection header received
0929 03:14:53.168908 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:53.168914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:53.168965 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:53.169052 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:53.169385 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:53.169970 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:53.170921 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:53.171315 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:53.171964 (+ 649us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:53.172543 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:53.172547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:53.172549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:53.172563 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:53.172591 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:53.172603 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:53.172696 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:53.172933 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 03:14:53.172968 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":42}
I20260929 03:14:53.174892 10174 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbb06f30c6854c7ebedc240126092f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb06f30c6854c7ebedc240126092f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } } }
I20260929 03:14:53.175025 10174 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:53.174645 10175 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb06f30c6854c7ebedc240126092f22. Latest consensus state: current_term: 1 leader_uuid: "dbb06f30c6854c7ebedc240126092f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb06f30c6854c7ebedc240126092f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } } }
I20260929 03:14:53.175339 10175 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22 [sys.catalog]: This master's current role is: LEADER
W20260929 03:14:53.177351 10187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:14:53.177543 10187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:14:53.177831 10188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:53.178612 10188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:53.183869 10188 catalog_manager.cc:1347] Generated new cluster ID: e44ee7f5b7434f05bcefec7b725c6285
I20260929 03:14:53.183993 10188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:53.212181 10188 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:53.213246 10188 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:53.223003 10188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbb06f30c6854c7ebedc240126092f22: Generated new TSK 0
I20260929 03:14:53.223791 10188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:53.231022 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 10095
W20260929 03:14:53.236773 10080 connection.cc:535] client connection to 127.0.0.1:41943 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41943): Connection reset by peer (error 104)
2026-09-29T03:14:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:53 dist-test-slave-k1dg krb5kdc[10194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:53 dist-test-slave-k1dg krb5kdc[10194](info): set up 2 sockets
Sep 29 03:14:53 dist-test-slave-k1dg krb5kdc[10194](info): commencing operation
krb5kdc: starting...
W20260929 03:14:54.286964 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:54 dist-test-slave-k1dg krb5kdc[10194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:14:54.415549 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:54.536563 10207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:54.536896 10207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:54.536981 10207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:54.541184 10207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:54.541296 10207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:54.541329 10207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:54.541371 10207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:54.541410 10207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:54.547014 10207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:54.548210 10207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:54.549279 10207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:54.558092 10212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:54.558213 10207 server_base.cc:1034] running on GCE node
W20260929 03:14:54.558218 10215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:54.558399 10213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:54.558947 10207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:54.560083 10207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:54.561275 10207 hybrid_clock.cc:648] HybridClock initialized: now 1790651694561251 us; error 37 us; skew 500 ppm
Sep 29 03:14:54 dist-test-slave-k1dg krb5kdc[10194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:54.564144 10207 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:14:54.565079 10207 webserver.cc:460] Webserver started at http://127.0.0.1:45599/ using document root <none> and password file <none>
I20260929 03:14:54.565599 10207 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:54.565657 10207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:54.565822 10207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:54.567919 10207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "aded11c691864ebdb0a648f26fc5d4b6"
format_stamp: "Formatted at 2026-09-29 03:14:54 on dist-test-slave-k1dg"
server_key: "fadf4e914ce08e356045087717277ca1"
server_key_iv: "7639901e7a2232fac0df14ce0a897853"
server_key_version: "encryptionkey@0"
I20260929 03:14:54.568439 10207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "aded11c691864ebdb0a648f26fc5d4b6"
format_stamp: "Formatted at 2026-09-29 03:14:54 on dist-test-slave-k1dg"
server_key: "fadf4e914ce08e356045087717277ca1"
server_key_iv: "7639901e7a2232fac0df14ce0a897853"
server_key_version: "encryptionkey@0"
I20260929 03:14:54.572161 10207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:14:54.574694 10222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:54.576069 10207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:54.576232 10207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "aded11c691864ebdb0a648f26fc5d4b6"
format_stamp: "Formatted at 2026-09-29 03:14:54 on dist-test-slave-k1dg"
server_key: "fadf4e914ce08e356045087717277ca1"
server_key_iv: "7639901e7a2232fac0df14ce0a897853"
server_key_version: "encryptionkey@0"
I20260929 03:14:54.576365 10207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:54.581000 10207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:54.583874 10207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:54.584066 10207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:54.591183 10207 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45495
I20260929 03:14:54.591210 10274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45495 every 8 connection(s)
I20260929 03:14:54.592159 10207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:54.592528 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 10207
I20260929 03:14:54.592691 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:54.592968 21177 external_mini_cluster.cc:1428] Setting key d0f564bb66caa41f4a6f225d3d0d568b
I20260929 03:14:54.595558 10275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:54.598765 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:54.594390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:60575 (local address 127.0.0.1:45495)
0929 03:14:54.594967 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:54.594990 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:54.595034 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:14:54.595881 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:54.595904 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:54.596249 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:54.596622 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:54.597329 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:54.597335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:54.597349 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:54.597360 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:54.597417 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:54.597447 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:54.597587 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:54.597652 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 03:14:54.597810 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":108,"threads_started":1}
I20260929 03:14:54.601673 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:54.602350 10275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6: Bootstrap starting.
I20260929 03:14:54.605005 10275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:54.605562 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:54.603801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:34997 (local address 127.0.0.1:45495)
0929 03:14:54.603939 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:54.603943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:54.604126 (+ 183us) server_negotiation.cc:407] Connection header received
0929 03:14:54.604250 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:54.604252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:54.604282 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:54.604354 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:54.605126 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:54.605129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:54.605132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:54.605149 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:54.605174 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:54.605184 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:54.605250 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:54.605381 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 03:14:54.605416 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:14:54.605957 10275 log.cc:824] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:54.607956 10275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6: No bootstrap required, opened a new log
I20260929 03:14:54.611613 10275 raft_consensus.cc:348] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded11c691864ebdb0a648f26fc5d4b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } }
I20260929 03:14:54.611877 10275 raft_consensus.cc:374] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:54.611959 10275 raft_consensus.cc:729] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aded11c691864ebdb0a648f26fc5d4b6, State: Initialized, Role: FOLLOWER
I20260929 03:14:54.612434 10275 consensus_queue.cc:260] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded11c691864ebdb0a648f26fc5d4b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } }
I20260929 03:14:54.612599 10275 raft_consensus.cc:388] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:54.612687 10275 raft_consensus.cc:482] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:54.612785 10275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:54.613716 10275 raft_consensus.cc:504] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded11c691864ebdb0a648f26fc5d4b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } }
I20260929 03:14:54.614049 10275 leader_election.cc:302] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aded11c691864ebdb0a648f26fc5d4b6; no voters:
I20260929 03:14:54.614367 10275 leader_election.cc:288] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:54.614509 10285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:54.614750 10285 raft_consensus.cc:686] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [term 1 LEADER]: Becoming Leader. State: Replica: aded11c691864ebdb0a648f26fc5d4b6, State: Running, Role: LEADER
I20260929 03:14:54.615062 10285 consensus_queue.cc:237] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded11c691864ebdb0a648f26fc5d4b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } }
I20260929 03:14:54.615648 10275 sys_catalog.cc:564] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:54.616647 10286 sys_catalog.cc:455] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aded11c691864ebdb0a648f26fc5d4b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded11c691864ebdb0a648f26fc5d4b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } } }
I20260929 03:14:54.616786 10286 sys_catalog.cc:458] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:54.617019 10287 sys_catalog.cc:455] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aded11c691864ebdb0a648f26fc5d4b6. Latest consensus state: current_term: 1 leader_uuid: "aded11c691864ebdb0a648f26fc5d4b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aded11c691864ebdb0a648f26fc5d4b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45495 } } }
I20260929 03:14:54.617116 10287 sys_catalog.cc:458] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:54.617270 10294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:54.620005 10294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:54.625389 10294 catalog_manager.cc:1347] Generated new cluster ID: d1d480a5b99a4c04944765fc07d2abe5
I20260929 03:14:54.625469 10294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:54.638163 10294 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:54.638567 10294 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:54.645750 10294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aded11c691864ebdb0a648f26fc5d4b6: Generated new TSK 0
I20260929 03:14:54.646549 10294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:54.657580 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 10207
2026-09-29T03:14:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:54 dist-test-slave-k1dg krb5kdc[10306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:54 dist-test-slave-k1dg krb5kdc[10306](info): set up 2 sockets
Sep 29 03:14:54 dist-test-slave-k1dg krb5kdc[10306](info): commencing operation
krb5kdc: starting...
W20260929 03:14:55.698835 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:55 dist-test-slave-k1dg krb5kdc[10306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:14:55.826128 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44751
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:44751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:55.938484 10319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:55.938834 10319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:55.938928 10319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:55.943051 10319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:55.943145 10319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:55.943183 10319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:55.943250 10319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:55.943291 10319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:55.948657 10319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:44751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:44751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:55.949770 10319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:55.950878 10319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:55.959731 10325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:55.959718 10327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:55.959694 10319 server_base.cc:1034] running on GCE node
W20260929 03:14:55.959690 10324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:55.960542 10319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:55.961733 10319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:55.962934 10319 hybrid_clock.cc:648] HybridClock initialized: now 1790651695962917 us; error 27 us; skew 500 ppm
Sep 29 03:14:55 dist-test-slave-k1dg krb5kdc[10306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:55.965910 10319 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:14:55.966939 10319 webserver.cc:460] Webserver started at http://127.20.174.126:45419/ using document root <none> and password file <none>
I20260929 03:14:55.967501 10319 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:55.967564 10319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:55.967818 10319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:55.969934 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "9a86fac78cfd4c1ca59a9dc7cede0bf6"
format_stamp: "Formatted at 2026-09-29 03:14:55 on dist-test-slave-k1dg"
server_key: "994efab9aa89abc85e4c27a066d51862"
server_key_iv: "edb3fce6f40adeadace0e7b3520996ac"
server_key_version: "encryptionkey@0"
I20260929 03:14:55.970539 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "9a86fac78cfd4c1ca59a9dc7cede0bf6"
format_stamp: "Formatted at 2026-09-29 03:14:55 on dist-test-slave-k1dg"
server_key: "994efab9aa89abc85e4c27a066d51862"
server_key_iv: "edb3fce6f40adeadace0e7b3520996ac"
server_key_version: "encryptionkey@0"
I20260929 03:14:55.974264 10319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:14:55.976954 10334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:55.978251 10319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:14:55.978416 10319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "9a86fac78cfd4c1ca59a9dc7cede0bf6"
format_stamp: "Formatted at 2026-09-29 03:14:55 on dist-test-slave-k1dg"
server_key: "994efab9aa89abc85e4c27a066d51862"
server_key_iv: "edb3fce6f40adeadace0e7b3520996ac"
server_key_version: "encryptionkey@0"
I20260929 03:14:55.978595 10319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:55.992102 10319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:55.995200 10319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:55.995414 10319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:56.002615 10319 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:44751
I20260929 03:14:56.002645 10386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:44751 every 8 connection(s)
I20260929 03:14:56.003609 10319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:56.006513 10387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:56.012184 10387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6: Bootstrap starting.
I20260929 03:14:56.012748 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 10319
I20260929 03:14:56.012907 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:56.013180 21177 external_mini_cluster.cc:1428] Setting key b364d09380a381e274660d8a4cff3248
I20260929 03:14:56.015262 10387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:56.016019 10387 log.cc:824] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:56.018342 10387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6: No bootstrap required, opened a new log
Sep 29 03:14:56 dist-test-slave-k1dg krb5kdc[10306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:14:56.023905 10387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a86fac78cfd4c1ca59a9dc7cede0bf6" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44751 } }
I20260929 03:14:56.024216 10387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:56.024286 10387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a86fac78cfd4c1ca59a9dc7cede0bf6, State: Initialized, Role: FOLLOWER
I20260929 03:14:56.024870 10387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a86fac78cfd4c1ca59a9dc7cede0bf6" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44751 } }
I20260929 03:14:56.025075 10387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:56.025173 10387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:56.025328 10387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:56.026721 10387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a86fac78cfd4c1ca59a9dc7cede0bf6" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44751 } }
I20260929 03:14:56.027143 10387 leader_election.cc:302] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a86fac78cfd4c1ca59a9dc7cede0bf6; no voters:
I20260929 03:14:56.027503 10387 leader_election.cc:288] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:56.027617 10392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:56.027981 10392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 9a86fac78cfd4c1ca59a9dc7cede0bf6, State: Running, Role: LEADER
I20260929 03:14:56.028327 10392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a86fac78cfd4c1ca59a9dc7cede0bf6" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44751 } }
I20260929 03:14:56.029068 10387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:56.029798 10392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a86fac78cfd4c1ca59a9dc7cede0bf6. Latest consensus state: current_term: 1 leader_uuid: "9a86fac78cfd4c1ca59a9dc7cede0bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a86fac78cfd4c1ca59a9dc7cede0bf6" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44751 } } }
I20260929 03:14:56.029925 10392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:56.030225 10393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a86fac78cfd4c1ca59a9dc7cede0bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a86fac78cfd4c1ca59a9dc7cede0bf6" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 44751 } } }
I20260929 03:14:56.030301 10393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:56.030580 10399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:56.030942 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:56.014713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35549 (local address 127.20.174.126:44751)
0929 03:14:56.015183 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:56.015199 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:56.015236 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:14:56.015965 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:56.015983 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:56.016434 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:56.016835 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:56.017812 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:56.018979 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:56.020425 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:56.020897 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:56.021039 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:56.024217 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:56.024226 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:56.024256 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:56.024299 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:56.026619 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:56.027434 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:56.027440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:56.027446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:56.027634 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:56.028128 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:56.028132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:56.028134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:56.028702 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:56.028947 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:56.029433 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 03:14:56.029711 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":149,"threads_started":1}
I20260929 03:14:56.033800 10399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:56.034689 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:56.042136 10399 catalog_manager.cc:1347] Generated new cluster ID: b7a8521ea1094c81a53a6f06f7e98bf1
I20260929 03:14:56.042232 10399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:56.047318 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:56.037382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41687 (local address 127.20.174.126:44751)
0929 03:14:56.037513 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:56.037518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:56.037807 (+ 289us) server_negotiation.cc:407] Connection header received
0929 03:14:56.037925 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:56.037929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:56.037990 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:56.038088 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:56.039112 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:56.040127 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:56.041058 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:56.041310 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:56.041424 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:56.043298 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:56.043303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:56.043306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:14:56.043338 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:56.045177 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:56.045744 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:56.045749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:56.045751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:56.045854 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:14:56.046217 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:14:56.046221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:14:56.046224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:14:56.046522 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:56.046730 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:56.046995 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 03:14:56.047113 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":53}
I20260929 03:14:56.071730 10399 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:56.073036 10399 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:56.084139 10399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a86fac78cfd4c1ca59a9dc7cede0bf6: Generated new TSK 0
I20260929 03:14:56.085058 10399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:56.090533 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 10319
W20260929 03:14:56.095844 10304 connection.cc:535] client connection to 127.20.174.126:44751 recv error: Network error: failed to read from TLS socket (remote: 127.20.174.126:44751): Connection reset by peer (error 104)
2026-09-29T03:14:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:56 dist-test-slave-k1dg krb5kdc[10418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:56 dist-test-slave-k1dg krb5kdc[10418](info): set up 2 sockets
Sep 29 03:14:56 dist-test-slave-k1dg krb5kdc[10418](info): commencing operation
krb5kdc: starting...
W20260929 03:14:57.154917 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:57 dist-test-slave-k1dg krb5kdc[10418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:14:57.282179 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:37975
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:33191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:37975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:57.394536 10431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:57.394827 10431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:57.394871 10431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:57.398921 10431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:57.398979 10431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:57.398999 10431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:57.399016 10431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:57.399082 10431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:57.404399 10431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:33191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:37975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:37975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:57.405467 10431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:57.406544 10431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:57.415572 10436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:57.415616 10437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:57.415572 10439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:57.415881 10431 server_base.cc:1034] running on GCE node
I20260929 03:14:57.416505 10431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:57.417655 10431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:57.418887 10431 hybrid_clock.cc:648] HybridClock initialized: now 1790651697418875 us; error 27 us; skew 500 ppm
Sep 29 03:14:57 dist-test-slave-k1dg krb5kdc[10418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651697, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:57.421656 10431 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:14:57.422617 10431 webserver.cc:460] Webserver started at http://127.20.174.126:43847/ using document root <none> and password file <none>
I20260929 03:14:57.423168 10431 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:57.423226 10431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:57.423468 10431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:57.425522 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "2058c917ddea4793bdc3397ca1653320"
format_stamp: "Formatted at 2026-09-29 03:14:57 on dist-test-slave-k1dg"
server_key: "17483c55c254089d60e11eee055206b2"
server_key_iv: "3c30ba1e99af4c8eb3697ed41a46c6bf"
server_key_version: "encryptionkey@0"
I20260929 03:14:57.426093 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "2058c917ddea4793bdc3397ca1653320"
format_stamp: "Formatted at 2026-09-29 03:14:57 on dist-test-slave-k1dg"
server_key: "17483c55c254089d60e11eee055206b2"
server_key_iv: "3c30ba1e99af4c8eb3697ed41a46c6bf"
server_key_version: "encryptionkey@0"
I20260929 03:14:57.429981 10431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:14:57.432474 10446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:57.433687 10431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:14:57.433874 10431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "2058c917ddea4793bdc3397ca1653320"
format_stamp: "Formatted at 2026-09-29 03:14:57 on dist-test-slave-k1dg"
server_key: "17483c55c254089d60e11eee055206b2"
server_key_iv: "3c30ba1e99af4c8eb3697ed41a46c6bf"
server_key_version: "encryptionkey@0"
I20260929 03:14:57.433976 10431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:57.456414 10431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:57.460062 10431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:57.460274 10431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:57.467676 10431 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:37975
I20260929 03:14:57.467828 10498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:37975 every 8 connection(s)
I20260929 03:14:57.468727 10431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:57.471902 10499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:57.477975 10499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320: Bootstrap starting.
I20260929 03:14:57.478946 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 10431
I20260929 03:14:57.479104 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:57.479442 21177 external_mini_cluster.cc:1428] Setting key 3d62167fe87e22b74acb34c42f782c98
I20260929 03:14:57.481251 10499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:57.482028 10499 log.cc:824] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:57.484438 10499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320: No bootstrap required, opened a new log
I20260929 03:14:57.489089 10499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2058c917ddea4793bdc3397ca1653320" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37975 } }
I20260929 03:14:57.489389 10499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:57.489483 10499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2058c917ddea4793bdc3397ca1653320, State: Initialized, Role: FOLLOWER
I20260929 03:14:57.489991 10499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2058c917ddea4793bdc3397ca1653320" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37975 } }
I20260929 03:14:57.490142 10499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:57.490191 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:57.480656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40259 (local address 127.20.174.126:37975)
0929 03:14:57.481159 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:57.481174 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:57.481213 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:14:57.481981 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:57.482000 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:57.482503 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:57.482861 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:57.483321 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:57.485072 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:57.486506 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:57.487016 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:57.487148 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:57.488172 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:57.488177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:57.488191 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:57.488621 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:57.488680 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:57.488704 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:57.488808 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:57.489026 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 03:14:57.489207 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":155,"threads_started":1}
I20260929 03:14:57.490221 10499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:57.490358 10499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:57.491560 10499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2058c917ddea4793bdc3397ca1653320" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37975 } }
I20260929 03:14:57.491921 10499 leader_election.cc:302] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2058c917ddea4793bdc3397ca1653320; no voters:
I20260929 03:14:57.492259 10499 leader_election.cc:288] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:57.492374 10504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:57.492619 10504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [term 1 LEADER]: Becoming Leader. State: Replica: 2058c917ddea4793bdc3397ca1653320, State: Running, Role: LEADER
I20260929 03:14:57.493010 10504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2058c917ddea4793bdc3397ca1653320" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37975 } }
I20260929 03:14:57.493481 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:57.493633 10499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:57.495152 10506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2058c917ddea4793bdc3397ca1653320. Latest consensus state: current_term: 1 leader_uuid: "2058c917ddea4793bdc3397ca1653320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2058c917ddea4793bdc3397ca1653320" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37975 } } }
I20260929 03:14:57.495308 10506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:57.496737 10505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2058c917ddea4793bdc3397ca1653320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2058c917ddea4793bdc3397ca1653320" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 37975 } } }
I20260929 03:14:57.496862 10505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320 [sys.catalog]: This master's current role is: LEADER
W20260929 03:14:57.499363 10523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:14:57.499446 10523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:14:57.499512 10517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:57.500219 10517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:57.501598 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:57.496994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57541 (local address 127.20.174.126:37975)
0929 03:14:57.497108 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:57.497113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:57.497419 (+ 306us) server_negotiation.cc:407] Connection header received
0929 03:14:57.497638 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:57.497641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:57.497702 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:57.497764 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:57.498222 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:57.498831 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:57.499764 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:14:57.500028 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:14:57.500141 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:14:57.501077 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:57.501081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:57.501083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:57.501094 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:57.501118 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:57.501129 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:57.501261 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:57.501420 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:14:57.501456 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:14:57.506520 10517 catalog_manager.cc:1347] Generated new cluster ID: e8f250dbff8e45aeb7a866231416d53c
I20260929 03:14:57.506610 10517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:57.531411 10517 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:57.532793 10517 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:57.538733 10517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2058c917ddea4793bdc3397ca1653320: Generated new TSK 0
I20260929 03:14:57.539482 10517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:57.546028 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 10431
W20260929 03:14:57.551625 10416 connection.cc:535] client connection to 127.20.174.126:37975 recv error: Network error: failed to read from TLS socket (remote: 127.20.174.126:37975): Connection reset by peer (error 104)
2026-09-29T03:14:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:57 dist-test-slave-k1dg krb5kdc[10530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:57 dist-test-slave-k1dg krb5kdc[10530](info): set up 2 sockets
Sep 29 03:14:57 dist-test-slave-k1dg krb5kdc[10530](info): commencing operation
krb5kdc: starting...
W20260929 03:14:58.598856 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:14:58 dist-test-slave-k1dg krb5kdc[10530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:14:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:14:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:14:58.727443 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:42793
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:35467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:42793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:14:58.841735 10543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:14:58.842095 10543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:14:58.842177 10543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:14:58.846870 10543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:14:58.846994 10543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:14:58.847038 10543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:14:58.847101 10543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:14:58.847150 10543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:14:58.852867 10543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:35467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:42793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:42793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:14:58.854082 10543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:14:58.855221 10543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:14:58.864359 10549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:14:58.864331 10548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:58.864480 10543 server_base.cc:1034] running on GCE node
W20260929 03:14:58.864333 10551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:14:58.865226 10543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:14:58.866425 10543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:14:58.867709 10543 hybrid_clock.cc:648] HybridClock initialized: now 1790651698867683 us; error 37 us; skew 500 ppm
Sep 29 03:14:58 dist-test-slave-k1dg krb5kdc[10530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651698, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:14:58.870615 10543 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:14:58.871626 10543 webserver.cc:460] Webserver started at http://127.20.174.126:36491/ using document root <none> and password file <none>
I20260929 03:14:58.872258 10543 fs_manager.cc:359] Metadata directory not provided
I20260929 03:14:58.872365 10543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:14:58.872610 10543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:14:58.875021 10543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "bd9e3b35cc694e76984b4d7ab959c02e"
format_stamp: "Formatted at 2026-09-29 03:14:58 on dist-test-slave-k1dg"
server_key: "1196b2e531caa186b7b896df80f7c2b8"
server_key_iv: "b35ebfb36713a1fd251271379c14773b"
server_key_version: "encryptionkey@0"
I20260929 03:14:58.875636 10543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "bd9e3b35cc694e76984b4d7ab959c02e"
format_stamp: "Formatted at 2026-09-29 03:14:58 on dist-test-slave-k1dg"
server_key: "1196b2e531caa186b7b896df80f7c2b8"
server_key_iv: "b35ebfb36713a1fd251271379c14773b"
server_key_version: "encryptionkey@0"
I20260929 03:14:58.879763 10543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 03:14:58.882551 10558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:14:58.883984 10543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:14:58.884176 10543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "bd9e3b35cc694e76984b4d7ab959c02e"
format_stamp: "Formatted at 2026-09-29 03:14:58 on dist-test-slave-k1dg"
server_key: "1196b2e531caa186b7b896df80f7c2b8"
server_key_iv: "b35ebfb36713a1fd251271379c14773b"
server_key_version: "encryptionkey@0"
I20260929 03:14:58.884351 10543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:14:58.889236 10543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:14:58.892283 10543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:14:58.892493 10543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:14:58.899976 10543 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:42793
I20260929 03:14:58.900004 10610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:42793 every 8 connection(s)
I20260929 03:14:58.901033 10543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:14:58.903970 10611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:14:58.904369 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 10543
I20260929 03:14:58.904510 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:14:58.904841 21177 external_mini_cluster.cc:1428] Setting key 3bbc98cf1be08bac9d92bcf5aadde892
I20260929 03:14:58.910399 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:58.906346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38267 (local address 127.20.174.126:42793)
0929 03:14:58.906804 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:58.906826 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:58.906864 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:14:58.907560 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:58.907578 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:58.908014 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:58.908372 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:58.909216 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:58.909221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:58.909233 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:58.909244 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:58.909296 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:58.909314 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:58.909462 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:58.909534 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:14:58.909695 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":126,"threads_started":1}
I20260929 03:14:58.911219 10611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e: Bootstrap starting.
I20260929 03:14:58.913599 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:14:58.914013 10611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e: Neither blocks nor log segments found. Creating new log.
I20260929 03:14:58.914832 10611 log.cc:824] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:14:58.917316 10611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e: No bootstrap required, opened a new log
I20260929 03:14:58.917440 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:14:58.915760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34499 (local address 127.20.174.126:42793)
0929 03:14:58.915899 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 03:14:58.915904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:14:58.916052 (+ 148us) server_negotiation.cc:407] Connection header received
0929 03:14:58.916162 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:14:58.916164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:14:58.916200 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:14:58.916277 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:14:58.917020 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:14:58.917023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:14:58.917026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:14:58.917038 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:14:58.917068 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:14:58.917083 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:14:58.917173 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:14:58.917234 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 03:14:58.917270 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:14:58.921142 10611 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9e3b35cc694e76984b4d7ab959c02e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42793 } }
I20260929 03:14:58.921406 10611 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:14:58.921443 10611 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9e3b35cc694e76984b4d7ab959c02e, State: Initialized, Role: FOLLOWER
I20260929 03:14:58.921844 10611 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9e3b35cc694e76984b4d7ab959c02e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42793 } }
I20260929 03:14:58.921972 10611 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:14:58.922012 10611 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:14:58.922116 10611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:14:58.923278 10611 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9e3b35cc694e76984b4d7ab959c02e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42793 } }
I20260929 03:14:58.923669 10611 leader_election.cc:302] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd9e3b35cc694e76984b4d7ab959c02e; no voters:
I20260929 03:14:58.924026 10611 leader_election.cc:288] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:14:58.924150 10621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:14:58.924415 10621 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [term 1 LEADER]: Becoming Leader. State: Replica: bd9e3b35cc694e76984b4d7ab959c02e, State: Running, Role: LEADER
I20260929 03:14:58.924769 10621 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9e3b35cc694e76984b4d7ab959c02e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42793 } }
I20260929 03:14:58.925426 10611 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:14:58.926800 10622 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd9e3b35cc694e76984b4d7ab959c02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9e3b35cc694e76984b4d7ab959c02e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42793 } } }
I20260929 03:14:58.926865 10623 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd9e3b35cc694e76984b4d7ab959c02e. Latest consensus state: current_term: 1 leader_uuid: "bd9e3b35cc694e76984b4d7ab959c02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9e3b35cc694e76984b4d7ab959c02e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42793 } } }
I20260929 03:14:58.926956 10622 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:58.926968 10623 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e [sys.catalog]: This master's current role is: LEADER
I20260929 03:14:58.927362 10631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:14:58.930393 10631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:14:58.936582 10631 catalog_manager.cc:1347] Generated new cluster ID: 367c04432efc4680a2e00655ce9b67c6
I20260929 03:14:58.936672 10631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:14:58.947141 10631 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:14:58.947562 10631 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:14:58.951558 10631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd9e3b35cc694e76984b4d7ab959c02e: Generated new TSK 0
I20260929 03:14:58.952458 10631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:14:58.960845 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 10543
2026-09-29T03:14:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:14:59 dist-test-slave-k1dg krb5kdc[10642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:14:59 dist-test-slave-k1dg krb5kdc[10642](info): set up 2 sockets
Sep 29 03:14:59 dist-test-slave-k1dg krb5kdc[10642](info): commencing operation
krb5kdc: starting...
W20260929 03:15:00.010924 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:15:00 dist-test-slave-k1dg krb5kdc[10642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:15:00.133482 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:43451
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:42203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:43451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:00.249202 10655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:00.249576 10655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:00.249662 10655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:00.254020 10655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:00.254169 10655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:00.254205 10655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:00.254245 10655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:00.254281 10655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:00.260079 10655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:42203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:43451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:43451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:00.261332 10655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:00.262503 10655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:00.271934 10663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:00.271935 10661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:00.271937 10660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:00.272455 10655 server_base.cc:1034] running on GCE node
I20260929 03:15:00.273074 10655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:00.274293 10655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:00.275504 10655 hybrid_clock.cc:648] HybridClock initialized: now 1790651700275488 us; error 28 us; skew 500 ppm
Sep 29 03:15:00 dist-test-slave-k1dg krb5kdc[10642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651700, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:00.278519 10655 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:15:00.279500 10655 webserver.cc:460] Webserver started at http://127.20.174.126:38451/ using document root <none> and password file <none>
I20260929 03:15:00.280051 10655 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:00.280112 10655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:00.280287 10655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:00.282595 10655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "342b6ccd9cc045ae8d58664a0dc4e9ca"
format_stamp: "Formatted at 2026-09-29 03:15:00 on dist-test-slave-k1dg"
server_key: "aac0970a506e4d4456fa6292454efebf"
server_key_iv: "650575869577042a5e373ef447db9965"
server_key_version: "encryptionkey@0"
I20260929 03:15:00.283166 10655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "342b6ccd9cc045ae8d58664a0dc4e9ca"
format_stamp: "Formatted at 2026-09-29 03:15:00 on dist-test-slave-k1dg"
server_key: "aac0970a506e4d4456fa6292454efebf"
server_key_iv: "650575869577042a5e373ef447db9965"
server_key_version: "encryptionkey@0"
I20260929 03:15:00.287180 10655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:15:00.289891 10670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:00.291469 10655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:15:00.291590 10655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "342b6ccd9cc045ae8d58664a0dc4e9ca"
format_stamp: "Formatted at 2026-09-29 03:15:00 on dist-test-slave-k1dg"
server_key: "aac0970a506e4d4456fa6292454efebf"
server_key_iv: "650575869577042a5e373ef447db9965"
server_key_version: "encryptionkey@0"
I20260929 03:15:00.291692 10655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:00.308777 10655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:00.312580 10655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:00.312767 10655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:00.320513 10655 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:43451
I20260929 03:15:00.320539 10722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:43451 every 8 connection(s)
I20260929 03:15:00.321553 10655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:00.324481 10723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:00.330404 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 10655
I20260929 03:15:00.330451 10723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca: Bootstrap starting.
I20260929 03:15:00.330600 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:15:00.330915 21177 external_mini_cluster.cc:1428] Setting key 80eabd207a44676e7cd048b86f64d495
I20260929 03:15:00.333269 10723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:00.334105 10723 log.cc:824] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:00.336109 10723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca: No bootstrap required, opened a new log
I20260929 03:15:00.339967 10723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b6ccd9cc045ae8d58664a0dc4e9ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43451 } }
I20260929 03:15:00.340215 10723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:00.340278 10723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342b6ccd9cc045ae8d58664a0dc4e9ca, State: Initialized, Role: FOLLOWER
Sep 29 03:15:00 dist-test-slave-k1dg krb5kdc[10642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:15:00.340863 10723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b6ccd9cc045ae8d58664a0dc4e9ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43451 } }
I20260929 03:15:00.341028 10723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:00.341120 10723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:00.341228 10723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:00.342584 10723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b6ccd9cc045ae8d58664a0dc4e9ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43451 } }
I20260929 03:15:00.342988 10723 leader_election.cc:302] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 342b6ccd9cc045ae8d58664a0dc4e9ca; no voters:
I20260929 03:15:00.343357 10723 leader_election.cc:288] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:00.343464 10728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:00.343776 10728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [term 1 LEADER]: Becoming Leader. State: Replica: 342b6ccd9cc045ae8d58664a0dc4e9ca, State: Running, Role: LEADER
I20260929 03:15:00.344141 10728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b6ccd9cc045ae8d58664a0dc4e9ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43451 } }
I20260929 03:15:00.344848 10723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:00.345407 10728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342b6ccd9cc045ae8d58664a0dc4e9ca. Latest consensus state: current_term: 1 leader_uuid: "342b6ccd9cc045ae8d58664a0dc4e9ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b6ccd9cc045ae8d58664a0dc4e9ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43451 } } }
I20260929 03:15:00.345605 10728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:00.345881 10729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "342b6ccd9cc045ae8d58664a0dc4e9ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342b6ccd9cc045ae8d58664a0dc4e9ca" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 43451 } } }
I20260929 03:15:00.346058 10729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:00.346259 10737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:00.347544 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:00.332364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34767 (local address 127.20.174.126:43451)
0929 03:15:00.332844 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:00.332859 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:00.332902 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:15:00.333732 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:00.333754 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:00.334136 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:00.334740 (+ 604us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:00.335685 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:00.336756 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:00.337694 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:00.338205 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:00.338385 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:00.341267 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:00.341273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:00.341287 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:00.341343 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:00.343455 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:00.344046 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:00.344052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:00.344058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:00.344199 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:00.344518 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:00.344521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:00.344523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:00.345070 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:00.345278 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:00.345992 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 03:15:00.346327 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":149,"threads_started":1}
I20260929 03:15:00.349732 10737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:00.350850 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:15:00.355973 10737 catalog_manager.cc:1347] Generated new cluster ID: 7a0195b4aa19442b8a61b87495350cd5
I20260929 03:15:00.356047 10737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:00.362177 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:00.352956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59985 (local address 127.20.174.126:43451)
0929 03:15:00.353217 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:00.353224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:00.353240 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:15:00.353417 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:00.353420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:00.353474 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:00.353558 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:00.354401 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:00.355183 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:00.356412 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:00.356750 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:00.356863 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:00.358694 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:00.358698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:00.358701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:00.358727 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:00.360395 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:00.360915 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:00.360918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:00.360920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:00.361004 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:00.361285 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:00.361288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:00.361289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:00.361497 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:00.361610 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:00.361894 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 03:15:00.362010 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260929 03:15:00.372630 10737 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:00.374203 10737 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:00.384509 10737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 342b6ccd9cc045ae8d58664a0dc4e9ca: Generated new TSK 0
I20260929 03:15:00.385468 10737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:15:00.394685 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 10655
2026-09-29T03:15:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:00 dist-test-slave-k1dg krb5kdc[10754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:00 dist-test-slave-k1dg krb5kdc[10754](info): set up 2 sockets
Sep 29 03:15:00 dist-test-slave-k1dg krb5kdc[10754](info): commencing operation
krb5kdc: starting...
W20260929 03:15:01.439081 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:15:01 dist-test-slave-k1dg krb5kdc[10754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:15:01.579221 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40935
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:01.702781 10767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:01.703140 10767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:01.703310 10767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:01.708240 10767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:01.708359 10767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:01.708397 10767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:01.708459 10767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:01.708519 10767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:01.714725 10767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:40935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:01.716022 10767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:01.717275 10767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:01.726794 10773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:01.726819 10772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:01.726786 10775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:01.727121 10767 server_base.cc:1034] running on GCE node
I20260929 03:15:01.728024 10767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:01.729316 10767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:01.730530 10767 hybrid_clock.cc:648] HybridClock initialized: now 1790651701730486 us; error 70 us; skew 500 ppm
Sep 29 03:15:01 dist-test-slave-k1dg krb5kdc[10754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651701, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:01.741369 10767 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:15:01.742583 10767 webserver.cc:460] Webserver started at http://127.20.174.126:37405/ using document root <none> and password file <none>
I20260929 03:15:01.743299 10767 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:01.743408 10767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:01.743672 10767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:01.746081 10767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "29539738e2694898a994893798c87c2a"
format_stamp: "Formatted at 2026-09-29 03:15:01 on dist-test-slave-k1dg"
server_key: "3b2bef386de04b346869715b64059e3f"
server_key_iv: "220c43f2b9ae41e5ad56d070596df269"
server_key_version: "encryptionkey@0"
I20260929 03:15:01.746743 10767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "29539738e2694898a994893798c87c2a"
format_stamp: "Formatted at 2026-09-29 03:15:01 on dist-test-slave-k1dg"
server_key: "3b2bef386de04b346869715b64059e3f"
server_key_iv: "220c43f2b9ae41e5ad56d070596df269"
server_key_version: "encryptionkey@0"
I20260929 03:15:01.751030 10767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:15:01.753957 10782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:01.755537 10767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:15:01.755712 10767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "29539738e2694898a994893798c87c2a"
format_stamp: "Formatted at 2026-09-29 03:15:01 on dist-test-slave-k1dg"
server_key: "3b2bef386de04b346869715b64059e3f"
server_key_iv: "220c43f2b9ae41e5ad56d070596df269"
server_key_version: "encryptionkey@0"
I20260929 03:15:01.755923 10767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:01.779835 10767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:01.783851 10767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:01.784088 10767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:01.791889 10767 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:40935
I20260929 03:15:01.791915 10834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:40935 every 8 connection(s)
I20260929 03:15:01.792989 10767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:01.796175 10835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:01.797137 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 10767
I20260929 03:15:01.797307 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:15:01.797636 21177 external_mini_cluster.cc:1428] Setting key 1101c51247ca611e42435b714e2fb415
I20260929 03:15:01.803071 10835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a: Bootstrap starting.
I20260929 03:15:01.805485 10835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:01.806279 10835 log.cc:824] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:01.807794 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:01.799225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51745 (local address 127.20.174.126:40935)
0929 03:15:01.799758 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:01.799773 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:01.799812 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:15:01.800494 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:01.800533 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:01.800943 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:01.801340 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:01.801780 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:01.802882 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:01.803939 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:01.804467 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:01.804659 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:01.805568 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:01.805573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:01.805589 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:01.806063 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:01.806123 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:01.806148 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:01.806268 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:01.806395 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:15:01.806662 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":154,"threads_started":1}
I20260929 03:15:01.808928 10835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a: No bootstrap required, opened a new log
I20260929 03:15:01.811082 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:15:01.813177 10835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29539738e2694898a994893798c87c2a" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40935 } }
I20260929 03:15:01.813547 10835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:01.813637 10835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29539738e2694898a994893798c87c2a, State: Initialized, Role: FOLLOWER
I20260929 03:15:01.814167 10835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29539738e2694898a994893798c87c2a" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40935 } }
I20260929 03:15:01.814337 10835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:01.814404 10835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:01.814532 10835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:01.815644 10835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29539738e2694898a994893798c87c2a" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40935 } }
I20260929 03:15:01.816040 10835 leader_election.cc:302] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29539738e2694898a994893798c87c2a; no voters:
I20260929 03:15:01.816448 10835 leader_election.cc:288] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:01.816563 10845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:01.816785 10845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [term 1 LEADER]: Becoming Leader. State: Replica: 29539738e2694898a994893798c87c2a, State: Running, Role: LEADER
I20260929 03:15:01.817165 10845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29539738e2694898a994893798c87c2a" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40935 } }
I20260929 03:15:01.817759 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:01.813266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34279 (local address 127.20.174.126:40935)
0929 03:15:01.813408 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:01.813412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:01.813683 (+ 271us) server_negotiation.cc:407] Connection header received
0929 03:15:01.813854 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:01.813857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:01.813911 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:01.814002 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:01.814285 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:01.814979 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:01.815881 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:01.816146 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:01.816263 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:01.817255 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:01.817259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:01.817262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:01.817276 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:01.817308 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:01.817321 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:01.817381 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:01.817564 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 03:15:01.817627 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:15:01.818025 10835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:01.819643 10846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29539738e2694898a994893798c87c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29539738e2694898a994893798c87c2a" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40935 } } }
I20260929 03:15:01.819799 10846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:01.820232 10847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29539738e2694898a994893798c87c2a. Latest consensus state: current_term: 1 leader_uuid: "29539738e2694898a994893798c87c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29539738e2694898a994893798c87c2a" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 40935 } } }
I20260929 03:15:01.820350 10847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a [sys.catalog]: This master's current role is: LEADER
W20260929 03:15:01.823503 10860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:01.823612 10860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:01.823681 10857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:01.824432 10857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:01.830888 10857 catalog_manager.cc:1347] Generated new cluster ID: 0ff25a0c4d6b4e349af67d88d3943afe
I20260929 03:15:01.830989 10857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:01.853811 10857 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:01.854931 10857 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:01.861923 10857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29539738e2694898a994893798c87c2a: Generated new TSK 0
I20260929 03:15:01.862776 10857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:15:01.872890 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 10767
2026-09-29T03:15:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:01 dist-test-slave-k1dg krb5kdc[10866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:01 dist-test-slave-k1dg krb5kdc[10866](info): set up 2 sockets
Sep 29 03:15:01 dist-test-slave-k1dg krb5kdc[10866](info): commencing operation
krb5kdc: starting...
W20260929 03:15:02.955011 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:15:03 dist-test-slave-k1dg krb5kdc[10866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:15:03.094513 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:46213
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:40355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:46213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:03.212320 10879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:03.212694 10879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:03.212783 10879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:03.217224 10879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:03.217344 10879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:03.217381 10879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:03.217418 10879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:03.217470 10879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:03.223397 10879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:40355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:46213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:46213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:03.224642 10879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:03.225795 10879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:03.235333 10887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:03.235342 10884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:03.235435 10885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:03.235589 10879 server_base.cc:1034] running on GCE node
I20260929 03:15:03.236441 10879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:03.237720 10879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:03.238888 10879 hybrid_clock.cc:648] HybridClock initialized: now 1790651703238853 us; error 51 us; skew 500 ppm
Sep 29 03:15:03 dist-test-slave-k1dg krb5kdc[10866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651703, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:03.241775 10879 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:15:03.242923 10879 webserver.cc:460] Webserver started at http://127.20.174.126:37375/ using document root <none> and password file <none>
I20260929 03:15:03.243537 10879 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:03.243602 10879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:03.243857 10879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:03.246081 10879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "502dc5a4053046f58dbda4e0b63253cf"
format_stamp: "Formatted at 2026-09-29 03:15:03 on dist-test-slave-k1dg"
server_key: "6ec1d63cdc7e990d392ef722362a8eb6"
server_key_iv: "b8a7ba9d1870fcca5f6bf4ff53b2fe68"
server_key_version: "encryptionkey@0"
I20260929 03:15:03.246719 10879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "502dc5a4053046f58dbda4e0b63253cf"
format_stamp: "Formatted at 2026-09-29 03:15:03 on dist-test-slave-k1dg"
server_key: "6ec1d63cdc7e990d392ef722362a8eb6"
server_key_iv: "b8a7ba9d1870fcca5f6bf4ff53b2fe68"
server_key_version: "encryptionkey@0"
I20260929 03:15:03.250917 10879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:15:03.253662 10894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:03.255203 10879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:15:03.255357 10879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "502dc5a4053046f58dbda4e0b63253cf"
format_stamp: "Formatted at 2026-09-29 03:15:03 on dist-test-slave-k1dg"
server_key: "6ec1d63cdc7e990d392ef722362a8eb6"
server_key_iv: "b8a7ba9d1870fcca5f6bf4ff53b2fe68"
server_key_version: "encryptionkey@0"
I20260929 03:15:03.255488 10879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:03.260579 10879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:03.264003 10879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:03.264233 10879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:03.272178 10879 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:46213
I20260929 03:15:03.272199 10946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:46213 every 8 connection(s)
I20260929 03:15:03.273239 10879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:03.276365 10947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:03.281347 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 10879
I20260929 03:15:03.281499 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:15:03.281806 21177 external_mini_cluster.cc:1428] Setting key 44ebfc16f654b3271304dd081c00a49c
I20260929 03:15:03.283064 10947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf: Bootstrap starting.
I20260929 03:15:03.285670 10947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:03.286583 10947 log.cc:824] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:03.287825 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:03.283449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56183 (local address 127.20.174.126:46213)
0929 03:15:03.283934 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:03.283950 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:03.283989 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:15:03.284672 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:03.284690 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:03.285002 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:03.285372 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:03.286269 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:03.286273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:03.286286 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:03.286296 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:03.286352 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:03.286372 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:03.286526 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:03.286667 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 03:15:03.286848 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":148,"threads_started":1}
I20260929 03:15:03.288878 10947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf: No bootstrap required, opened a new log
I20260929 03:15:03.291139 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:15:03.293797 10947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502dc5a4053046f58dbda4e0b63253cf" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46213 } }
I20260929 03:15:03.294145 10947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:03.294236 10947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502dc5a4053046f58dbda4e0b63253cf, State: Initialized, Role: FOLLOWER
I20260929 03:15:03.294798 10947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502dc5a4053046f58dbda4e0b63253cf" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46213 } }
I20260929 03:15:03.294976 10947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:03.295055 10947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:03.295162 10947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:03.295468 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:03.293613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38411 (local address 127.20.174.126:46213)
0929 03:15:03.293752 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:03.293756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:03.293969 (+ 213us) server_negotiation.cc:407] Connection header received
0929 03:15:03.294100 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:03.294102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:03.294165 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:03.294270 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:03.295020 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:03.295024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:03.295026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:03.295039 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:03.295069 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:03.295082 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:03.295200 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:03.295279 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:15:03.295318 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:15:03.296401 10947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502dc5a4053046f58dbda4e0b63253cf" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46213 } }
I20260929 03:15:03.296772 10947 leader_election.cc:302] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502dc5a4053046f58dbda4e0b63253cf; no voters:
I20260929 03:15:03.297082 10947 leader_election.cc:288] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:03.297205 10957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:03.297461 10957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [term 1 LEADER]: Becoming Leader. State: Replica: 502dc5a4053046f58dbda4e0b63253cf, State: Running, Role: LEADER
I20260929 03:15:03.297812 10957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502dc5a4053046f58dbda4e0b63253cf" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46213 } }
I20260929 03:15:03.298478 10947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:03.299760 10959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "502dc5a4053046f58dbda4e0b63253cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502dc5a4053046f58dbda4e0b63253cf" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46213 } } }
I20260929 03:15:03.299760 10958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 502dc5a4053046f58dbda4e0b63253cf. Latest consensus state: current_term: 1 leader_uuid: "502dc5a4053046f58dbda4e0b63253cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502dc5a4053046f58dbda4e0b63253cf" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 46213 } } }
I20260929 03:15:03.299916 10959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:03.299916 10958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:03.300333 10967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:03.304090 10967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:03.310496 10967 catalog_manager.cc:1347] Generated new cluster ID: 5217a4384f8d4fb7ab915324b6f3a800
I20260929 03:15:03.310631 10967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:03.326079 10967 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:03.326534 10967 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:03.330374 10967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 502dc5a4053046f58dbda4e0b63253cf: Generated new TSK 0
I20260929 03:15:03.331175 10967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:15:03.335333 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 10879
2026-09-29T03:15:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1461 ms)
[----------] 18 tests from AuthTokenIssuingTest (26308 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:03 dist-test-slave-k1dg krb5kdc[10978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:03 dist-test-slave-k1dg krb5kdc[10978](info): set up 2 sockets
Sep 29 03:15:03 dist-test-slave-k1dg krb5kdc[10978](info): commencing operation
krb5kdc: starting...
W20260929 03:15:04.458920 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:15:04 dist-test-slave-k1dg krb5kdc[10978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:15:04.590240 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:42123
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:42123,127.20.174.125:45763,127.20.174.124:33591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:04.708786 10991 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:15:04.709197 10991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:04.709249 10991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:04.709509 10991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:04.714834 10991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:04.714943 10991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:04.714985 10991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:04.715054 10991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:04.715116 10991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:04.721640 10991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:42123,127.20.174.125:45763,127.20.174.124:33591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:04.723083 10991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:04.724291 10991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:04.734833 10996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:04.734874 10991 server_base.cc:1034] running on GCE node
W20260929 03:15:04.734800 10997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:04.734830 10999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:04.735651 10991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:04.736948 10991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:04.738171 10991 hybrid_clock.cc:648] HybridClock initialized: now 1790651704738137 us; error 48 us; skew 500 ppm
Sep 29 03:15:04 dist-test-slave-k1dg krb5kdc[10978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:04.741309 10991 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:15:04.742334 10991 webserver.cc:460] Webserver started at http://127.20.174.126:40471/ using document root <none> and password file <none>
I20260929 03:15:04.743007 10991 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:04.743112 10991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:04.743358 10991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:04.745553 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "bde451a318c14fd481e1edad0ac015d8"
format_stamp: "Formatted at 2026-09-29 03:15:04 on dist-test-slave-k1dg"
server_key: "b5bdfde3752a337d4c5e446aef7b3ef1"
server_key_iv: "881de11b6da26f38ac90c3fe6823be46"
server_key_version: "encryptionkey@0"
I20260929 03:15:04.746181 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "bde451a318c14fd481e1edad0ac015d8"
format_stamp: "Formatted at 2026-09-29 03:15:04 on dist-test-slave-k1dg"
server_key: "b5bdfde3752a337d4c5e446aef7b3ef1"
server_key_iv: "881de11b6da26f38ac90c3fe6823be46"
server_key_version: "encryptionkey@0"
I20260929 03:15:04.750308 10991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:15:04.752928 11006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:04.754251 10991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:04.754410 10991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "bde451a318c14fd481e1edad0ac015d8"
format_stamp: "Formatted at 2026-09-29 03:15:04 on dist-test-slave-k1dg"
server_key: "b5bdfde3752a337d4c5e446aef7b3ef1"
server_key_iv: "881de11b6da26f38ac90c3fe6823be46"
server_key_version: "encryptionkey@0"
I20260929 03:15:04.754608 10991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:04.776360 10991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:04.780037 10991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:04.780272 10991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:04.788152 10991 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:42123
I20260929 03:15:04.788499 11058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:42123 every 8 connection(s)
I20260929 03:15:04.789472 10991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:04.792748 11059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:04.796717 11059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:04.797551 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 10991
I20260929 03:15:04.797690 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:15:04.797996 21177 external_mini_cluster.cc:1428] Setting key 9f97d7c95f00195766746e40c55114db
WARNING: no policy specified for kudu/127.20.174.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.125@KRBTEST.COM" created.
Sep 29 03:15:04 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:15:04.818012 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:04.800378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.20.174.126:42123)
0929 03:15:04.800677 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:04.800686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:04.800724 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:15:04.801659 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:04.801664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:04.801863 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:04.802025 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:04.803355 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:04.804247 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:04.805965 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:04.806350 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:04.806550 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:04.812279 (+ 5729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:04.812290 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:04.812316 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:04.812353 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:04.814330 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:04.815061 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:04.815067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:04.815068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:04.815200 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:04.815647 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:04.815650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:04.815652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:04.816093 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:04.816211 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:04.816776 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 03:15:04.817110 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":195,"thread_start_us":105,"threads_started":1}
I20260929 03:15:04.818014 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:04.799550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:42123 (local address 127.0.0.1:37114)
0929 03:15:04.800167 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:04.800240 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:04.801339 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:04.802086 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:04.802095 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:04.802712 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:04.803142 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:04.803168 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:04.804406 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:04.804412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:04.805819 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:04.805834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:04.806569 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:04.806574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:04.806733 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:04.807355 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:04.807378 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:04.812057 (+ 4679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:04.814516 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:04.814531 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:04.814547 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:04.814911 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:04.815321 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:04.815325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:04.815327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:04.815511 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:04.816232 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:04.816244 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:04.816466 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 03:15:04.816691 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 03:15:04.817217 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":117,"thread_start_us":138,"threads_started":1}
I20260929 03:15:04.820937 11059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:04.821802 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:04.821531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:45763 (local address 127.0.0.1:39300)
0929 03:15:04.821657 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:04.821733 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 03:15:04.822072 11009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.174.125:45763: connect: Connection refused (error 111)
I20260929 03:15:04.823066 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:04.822784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:45763 (local address 127.0.0.1:39302)
0929 03:15:04.822915 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:04.822979 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 03:15:04.823362 11059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.125:45763: Network error: Client connection negotiation failed: client connection to 127.20.174.125:45763: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab.
Entry for principal kudu/127.20.174.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab.
I20260929 03:15:04.831130 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.125:45763
--webserver_interface=127.20.174.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab
--principal=kudu/127.20.174.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:42123,127.20.174.125:45763,127.20.174.124:33591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:15:04.872681 11059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } attempt: 1
I20260929 03:15:04.873772 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:04.873444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:45763 (local address 127.0.0.1:39316)
0929 03:15:04.873620 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:04.873689 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 03:15:04.874661 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:04.874369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:45763 (local address 127.0.0.1:39326)
0929 03:15:04.874534 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:04.874601 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 03:15:04.874801 11059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.125:45763: Network error: Client connection negotiation failed: client connection to 127.20.174.125:45763: connect: Connection refused (error 111)
I20260929 03:15:04.942998 11059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } attempt: 2
I20260929 03:15:04.944013 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:04.943658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:45763 (local address 127.0.0.1:39332)
0929 03:15:04.943861 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:04.943932 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 03:15:04.944936 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:04.944632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:45763 (local address 127.0.0.1:39342)
0929 03:15:04.944820 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:04.944871 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:45763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260929 03:15:04.945083 11059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.125:45763: Network error: Client connection negotiation failed: client connection to 127.20.174.125:45763: connect: Connection refused (error 111)
W20260929 03:15:04.951712 11066 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:15:04.952111 11066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:04.952174 11066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:04.952256 11066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:04.956602 11066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:04.956705 11066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:04.956746 11066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:04.956801 11066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:04.956839 11066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:04.963110 11066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:42123,127.20.174.125:45763,127.20.174.124:33591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab
--principal=kudu/127.20.174.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.125:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.174.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:04.964370 11066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:04.965500 11066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:04.975485 11071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:04.975912 11066 server_base.cc:1034] running on GCE node
W20260929 03:15:04.975555 11074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:04.975543 11072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:04.976837 11066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:04.978930 11066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:04.980213 11066 hybrid_clock.cc:648] HybridClock initialized: now 1790651704980171 us; error 65 us; skew 500 ppm
Sep 29 03:15:04 dist-test-slave-k1dg krb5kdc[10978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:04.983623 11066 init.cc:377] Logged in from keytab as kudu/127.20.174.125@KRBTEST.COM (short username kudu)
I20260929 03:15:04.984760 11066 webserver.cc:460] Webserver started at http://127.20.174.125:44079/ using document root <none> and password file <none>
I20260929 03:15:04.985354 11066 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:04.985421 11066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:04.985623 11066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:04.988032 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/data/instance:
uuid: "077f9da7e2a148f2a16d2741d7313d9d"
format_stamp: "Formatted at 2026-09-29 03:15:04 on dist-test-slave-k1dg"
server_key: "903a744ba95e40cd61c01f8be525b82d"
server_key_iv: "0ccd3de3f4a6cfd0f14787d2cf5d4645"
server_key_version: "encryptionkey@0"
I20260929 03:15:04.988610 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/wal/instance:
uuid: "077f9da7e2a148f2a16d2741d7313d9d"
format_stamp: "Formatted at 2026-09-29 03:15:04 on dist-test-slave-k1dg"
server_key: "903a744ba95e40cd61c01f8be525b82d"
server_key_iv: "0ccd3de3f4a6cfd0f14787d2cf5d4645"
server_key_version: "encryptionkey@0"
I20260929 03:15:04.993368 11066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:15:04.996409 11081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:04.998028 11066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:15:04.998164 11066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/wal
uuid: "077f9da7e2a148f2a16d2741d7313d9d"
format_stamp: "Formatted at 2026-09-29 03:15:04 on dist-test-slave-k1dg"
server_key: "903a744ba95e40cd61c01f8be525b82d"
server_key_iv: "0ccd3de3f4a6cfd0f14787d2cf5d4645"
server_key_version: "encryptionkey@0"
I20260929 03:15:04.998276 11066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:05.014549 11066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:05.018342 11066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:05.018640 11066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:05.026536 11066 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.125:45763
I20260929 03:15:05.026820 11133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.125:45763 every 8 connection(s)
I20260929 03:15:05.027688 11066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
I20260929 03:15:05.028112 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 11066
I20260929 03:15:05.028251 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-1/wal/instance
I20260929 03:15:05.028550 21177 external_mini_cluster.cc:1428] Setting key ba105e6183746ae74bea35a1cf0f9207
I20260929 03:15:05.031083 11134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:05.035346 11134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.174.124@KRBTEST.COM; defaulting to no policy
I20260929 03:15:05.036247 11059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.124@KRBTEST.COM" created.
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:15:05.052476 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.037648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.20.174.126:42123)
0929 03:15:05.037782 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.037787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.038462 (+ 675us) server_negotiation.cc:407] Connection header received
0929 03:15:05.039505 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.039508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.039573 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.039654 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.040945 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.041722 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.043311 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.044401 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.044529 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.048129 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.048136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.048141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.048174 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.050026 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.050696 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.050699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.050701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.050865 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.051185 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.051188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.051189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.051439 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.051548 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.052054 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.052231 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:15:05.052515 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.036838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:45763 (local address 127.0.0.1:39344)
0929 03:15:05.037046 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.037066 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:05.037211 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:05.039649 (+ 2438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:05.039655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:05.039901 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:05.040023 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.040029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.042048 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.042054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.042732 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.042746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.043586 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.043590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.043701 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.044279 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:05.044326 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:05.046375 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:05.049825 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.049828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.049829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.050135 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.050667 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.050673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.050678 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.050793 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.051900 (+ 1107us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:05.051904 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:05.052037 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 03:15:05.052161 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 03:15:05.052325 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"spinlock_wait_cycles":6528}
I20260929 03:15:05.053444 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.037740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:42123 (local address 127.0.0.1:37118)
0929 03:15:05.038300 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.038344 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:05.039183 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:05.039762 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:05.039772 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:05.040364 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:05.040757 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.040775 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.041869 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.041874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.043172 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.043182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.044586 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.044592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.044706 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.045304 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:05.045322 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:05.047953 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:05.050134 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.050145 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.050160 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.050558 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.050978 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.050980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.050982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.051075 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.051580 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:05.051588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:05.051743 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 03:15:05.051984 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 03:15:05.052370 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":26,"spinlock_wait_cycles":8576,"thread_start_us":181,"threads_started":1}
I20260929 03:15:05.053452 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.037740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.20.174.125:45763)
0929 03:15:05.038189 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.038207 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.038250 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:15:05.039136 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.039159 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.039436 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.040171 (+ 735us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.040528 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.041928 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.042854 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.043436 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.043622 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.046584 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.046591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.046602 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.046626 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.049668 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.050286 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.050292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.050293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.050510 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.050905 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.050911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.050915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.051578 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.051942 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.052248 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.052462 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":310,"thread_start_us":134,"threads_started":1}
I20260929 03:15:05.054986 11134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:05.055763 11059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:05.056571 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.056275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55706)
0929 03:15:05.056458 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.056508 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 03:15:05.057355 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.057065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55714)
0929 03:15:05.057235 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.057285 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 03:15:05.057574 11059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:33591: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab.
Entry for principal kudu/127.20.174.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab.
I20260929 03:15:05.062011 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.124:33591
--webserver_interface=127.20.174.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab
--principal=kudu/127.20.174.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:39883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:42123,127.20.174.125:45763,127.20.174.124:33591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:15:05.064733 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.055940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.20.174.125:45763)
0929 03:15:05.056064 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.056070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.056087 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:15:05.056121 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.056124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.056170 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.056239 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.056755 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.057473 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.058353 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.058687 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.058826 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.061117 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.061121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.061124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.061152 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.062927 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.063358 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.063364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.063368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.063468 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.063792 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.063795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.063796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.063990 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.064069 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.064400 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.064544 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":49}
I20260929 03:15:05.064795 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.055748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:45763 (local address 127.0.0.1:39354)
0929 03:15:05.055890 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.055907 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:05.055999 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:05.056293 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:05.056300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:05.056506 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:05.056639 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.056647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.057588 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.057591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.058233 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.058242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.058825 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.058829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.058921 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.059470 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:05.059484 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:05.060931 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:05.063043 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.063047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.063049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.063246 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.063580 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.063585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.063589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.063694 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.064085 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:05.064089 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:05.064205 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 03:15:05.064323 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:15:05.064492 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 03:15:05.065758 11134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:05.066665 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.066294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55726)
0929 03:15:05.066502 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.066598 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 03:15:05.067222 11083 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
I20260929 03:15:05.068229 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.067997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55728)
0929 03:15:05.068117 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.068168 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 03:15:05.068408 11134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:33591: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
I20260929 03:15:05.088766 11059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } attempt: 1
I20260929 03:15:05.089720 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.089417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55742)
0929 03:15:05.089578 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.089641 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 03:15:05.090557 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.090304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55746)
0929 03:15:05.090401 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.090500 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 03:15:05.090695 11059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:33591: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
I20260929 03:15:05.117714 11134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } attempt: 1
I20260929 03:15:05.118839 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.118460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55760)
0929 03:15:05.118616 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.118684 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 03:15:05.119592 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.119357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55770)
0929 03:15:05.119475 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.119534 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 03:15:05.119758 11134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:33591: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
I20260929 03:15:05.165927 11059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } attempt: 2
I20260929 03:15:05.167029 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.166675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55780)
0929 03:15:05.166879 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.166945 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 03:15:05.167807 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.167574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55790)
0929 03:15:05.167696 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.167752 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 03:15:05.167948 11059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:33591: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
W20260929 03:15:05.184274 11140 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:15:05.184664 11140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:05.184743 11140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:05.184844 11140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 03:15:05.187922 11134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } attempt: 2
I20260929 03:15:05.188784 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.188485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55806)
0929 03:15:05.188635 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.188707 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 03:15:05.189383 11140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:05.189463 11140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:05.189510 11140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:05.189548 11140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260929 03:15:05.189568 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.189294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55818)
0929 03:15:05.189422 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.189497 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 03:15:05.189600 11140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:15:05.189743 11134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:33591: Network error: Client connection negotiation failed: client connection to 127.20.174.124:33591: connect: Connection refused (error 111)
I20260929 03:15:05.195655 11140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:39883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:42123,127.20.174.125:45763,127.20.174.124:33591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab
--principal=kudu/127.20.174.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.124:33591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.174.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:05.196872 11140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:05.198005 11140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:05.208422 11149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:05.208432 11146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:05.208525 11140 server_base.cc:1034] running on GCE node
W20260929 03:15:05.208452 11147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:05.209487 11140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:05.210940 11140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:05.212131 11140 hybrid_clock.cc:648] HybridClock initialized: now 1790651705212111 us; error 40 us; skew 500 ppm
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651705, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:05.215287 11140 init.cc:377] Logged in from keytab as kudu/127.20.174.124@KRBTEST.COM (short username kudu)
I20260929 03:15:05.216352 11140 webserver.cc:460] Webserver started at http://127.20.174.124:39741/ using document root <none> and password file <none>
I20260929 03:15:05.216982 11140 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:05.217079 11140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:05.217331 11140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:05.219630 11140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/data/instance:
uuid: "55088875ed9f4f69bc81cf763a23e3e4"
format_stamp: "Formatted at 2026-09-29 03:15:05 on dist-test-slave-k1dg"
server_key: "b5a33417151070526d160d14bfac7f33"
server_key_iv: "9d7319444824bcbcaa15f58c776a86fa"
server_key_version: "encryptionkey@0"
I20260929 03:15:05.220602 11140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/wal/instance:
uuid: "55088875ed9f4f69bc81cf763a23e3e4"
format_stamp: "Formatted at 2026-09-29 03:15:05 on dist-test-slave-k1dg"
server_key: "b5a33417151070526d160d14bfac7f33"
server_key_iv: "9d7319444824bcbcaa15f58c776a86fa"
server_key_version: "encryptionkey@0"
I20260929 03:15:05.225476 11140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:15:05.228384 11156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:05.229954 11140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:15:05.230165 11140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/wal
uuid: "55088875ed9f4f69bc81cf763a23e3e4"
format_stamp: "Formatted at 2026-09-29 03:15:05 on dist-test-slave-k1dg"
server_key: "b5a33417151070526d160d14bfac7f33"
server_key_iv: "9d7319444824bcbcaa15f58c776a86fa"
server_key_version: "encryptionkey@0"
I20260929 03:15:05.230304 11140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:05.241318 11140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:05.244809 11140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:05.244995 11140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:05.252801 11140 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.124:33591
I20260929 03:15:05.253142 11208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.124:33591 every 8 connection(s)
I20260929 03:15:05.253895 11140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
I20260929 03:15:05.257097 11209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:05.260056 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 11140
I20260929 03:15:05.260185 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651413504048-21177-0/minicluster-data/master-2/wal/instance
I20260929 03:15:05.260488 21177 external_mini_cluster.cc:1428] Setting key 9f891e3d3f3a5a78473c273e95865519
I20260929 03:15:05.261932 11209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:05.267115 11059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } attempt: 3
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651705, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
I20260929 03:15:05.276754 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.263880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.20.174.126:42123)
0929 03:15:05.264016 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.264021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.264401 (+ 380us) server_negotiation.cc:407] Connection header received
0929 03:15:05.264624 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.264628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.264679 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.264776 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.265600 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.266329 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.267810 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.268132 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.268213 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.271410 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.271414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.271417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.271450 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.273479 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.274197 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.274202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.274205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.274303 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.275333 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.275339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.275341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.275726 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.275818 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.276401 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.276546 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:15:05.277680 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.261981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.20.174.124:33591)
0929 03:15:05.262412 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.262488 (+ 76us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.262540 (+ 52us) server_negotiation.cc:407] Connection header received
0929 03:15:05.263220 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.263240 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.263592 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.264018 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.264947 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.266227 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.267590 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.268272 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.268423 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.271944 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.271949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.271961 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.271982 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.274299 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.274942 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.274948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.274949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.275084 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.275443 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.275447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.275448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.275934 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.276153 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.276476 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.276828 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":283,"spinlock_wait_cycles":2432,"thread_start_us":145,"threads_started":1}
I20260929 03:15:05.277684 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.263728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:42123 (local address 127.0.0.1:37126)
0929 03:15:05.264271 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.264324 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:05.264489 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:05.264904 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:05.264909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:05.265241 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:05.265407 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.265417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.266498 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.266502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.267679 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.267688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.268424 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.268427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.268570 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.269272 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:05.269289 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:05.271199 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:05.273600 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.273610 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.273628 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.274075 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.274949 (+ 874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.274953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.274956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.275183 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.275914 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:05.275922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:05.276100 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 03:15:05.276977 (+ 877us) client_negotiation.cc:240] Negotiation successful
0929 03:15:05.277181 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":4,"thread_start_us":141,"threads_started":1}
I20260929 03:15:05.278018 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.267577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55840)
0929 03:15:05.267794 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.267811 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:05.267896 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:05.268283 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:05.268286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:05.268487 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:05.268579 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.268586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.269666 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.269670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.270363 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.270374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.271111 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.271115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.271208 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.271711 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:05.271730 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:05.273342 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:05.276092 (+ 2750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.276095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.276097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.276373 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.276686 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.276689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.276691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.276766 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.277528 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:05.277532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:05.277628 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 03:15:05.277715 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:15:05.277860 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"spinlock_wait_cycles":4992}
I20260929 03:15:05.279112 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.267628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.20.174.124:33591)
0929 03:15:05.268081 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.268087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.268097 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:15:05.268122 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.268124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.268156 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.268268 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.268720 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.269525 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.270578 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.270967 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.271085 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.273491 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.273495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.273497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.273522 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.275970 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.276493 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.276498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.276502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.276585 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.277175 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.277178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.277179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.277410 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.278540 (+ 1130us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.278631 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.278759 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":121,"threads_started":1}
I20260929 03:15:05.279561 11209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:05.280921 11134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } attempt: 3
I20260929 03:15:05.282864 11059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8: Bootstrap starting.
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651705, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
I20260929 03:15:05.285961 11059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8: Neither blocks nor log segments found. Creating new log.
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
I20260929 03:15:05.287223 11059 log.cc:824] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:05.289881 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.280240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.20.174.125:45763)
0929 03:15:05.280358 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.280364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.280425 (+ 61us) server_negotiation.cc:407] Connection header received
0929 03:15:05.280550 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.280553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.280601 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.280704 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.281258 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.282042 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.283098 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.283401 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.283505 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.285946 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.285950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.285953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.285978 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.287815 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.288350 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.288355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.288357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.288446 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.288796 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.288801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.288804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.289134 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.289235 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.289596 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.289717 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:15:05.290328 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.280149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:45763 (local address 127.0.0.1:39370)
0929 03:15:05.280357 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.280372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:05.280466 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:05.280714 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:05.280716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:05.280921 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:05.281114 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.281124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.282148 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.282153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.282932 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.282960 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.283526 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.283530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.283630 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.284200 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:05.284218 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:05.285797 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:05.287907 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.287913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.287915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.288224 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.288575 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.288580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.288583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.288674 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.289282 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:05.289285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:05.289394 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 03:15:05.290033 (+ 639us) client_negotiation.cc:240] Negotiation successful
0929 03:15:05.290182 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260929 03:15:05.291050 11059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8: No bootstrap required, opened a new log
I20260929 03:15:05.291229 11209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:05.291630 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.281354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55856)
0929 03:15:05.281511 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.281549 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:05.281623 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:05.281922 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:05.281925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:05.282106 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:05.282200 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.282206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.283274 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.283278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.283988 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.283999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.284542 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.284546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.284632 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.285086 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:05.285102 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:05.286893 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:05.288851 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.288855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.288857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.289172 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.289491 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.289494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.289496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.289581 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.290263 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:05.290266 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:05.290363 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 03:15:05.290873 (+ 510us) client_negotiation.cc:240] Negotiation successful
0929 03:15:05.291023 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 03:15:05.292307 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.281583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.20.174.124:33591)
0929 03:15:05.281688 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.281693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.281706 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:05.281740 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.281742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.281791 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.282308 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.282396 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.283164 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.284129 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.284411 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.284541 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.287052 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.287056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.287058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.287085 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.288768 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.289283 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.289286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.289288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.289386 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.289684 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.289687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.289689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.289889 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.291913 (+ 2024us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.291988 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.292123 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":768}
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:15:05.295866 11134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d: Bootstrap starting.
I20260929 03:15:05.295893 11059 raft_consensus.cc:348] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } }
I20260929 03:15:05.296178 11059 raft_consensus.cc:729] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde451a318c14fd481e1edad0ac015d8, State: Initialized, Role: FOLLOWER
I20260929 03:15:05.296558 11059 consensus_queue.cc:260] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } }
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651705, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
I20260929 03:15:05.298161 11059 sys_catalog.cc:564] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:05.299188 11134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:05.300058 11134 log.cc:824] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:05.300458 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.286568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.20.174.126:42123)
0929 03:15:05.286686 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.286691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.287957 (+ 1266us) server_negotiation.cc:407] Connection header received
0929 03:15:05.288150 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.288153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.288203 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.288259 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.289185 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.289992 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.291279 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.291585 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.291676 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.294874 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.294878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.294881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.294911 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.296978 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.297564 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.297568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.297569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.297657 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.297972 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.297976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.297977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.298191 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.298655 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.300151 (+ 1496us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.300299 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:15:05.301638 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.291960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.20.174.124:33591)
0929 03:15:05.292062 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.292066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.292608 (+ 542us) server_negotiation.cc:407] Connection header received
0929 03:15:05.292739 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.292741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.292773 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.292831 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.293320 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.294066 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.295254 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.295556 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.295628 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.298108 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.298112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.298114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.298136 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.299869 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.300360 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.300364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.300366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.300465 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.300750 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.300753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.300755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.300970 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.301046 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.301355 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.301478 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:15:05.302042 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.291750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:33591 (local address 127.0.0.1:55862)
0929 03:15:05.292544 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:05.292560 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:05.292658 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:05.292954 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:05.292957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:05.293127 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:05.293210 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.293218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.294264 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.294267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.295129 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:05.295143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.295754 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.295757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:05.295840 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.296325 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:05.296340 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:05.297974 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:05.299998 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.300001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.300003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.300250 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.300573 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:05.300575 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:05.300577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:05.300659 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:05.301106 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:05.301108 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:05.301203 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 03:15:05.301767 (+ 564us) client_negotiation.cc:240] Negotiation successful
0929 03:15:05.301901 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556}
I20260929 03:15:05.303504 11216 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } } }
I20260929 03:15:05.303656 11216 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:05.305066 11209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4: Bootstrap starting.
I20260929 03:15:05.307093 11134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d: No bootstrap required, opened a new log
I20260929 03:15:05.304167 10991 master_runner.cc:186] Error getting master registration for 127.20.174.125:45763: OK, instance_id { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" instance_seqno: 1790651705024265 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:15:05.307592 10991 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 03:15:05.309844 11228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:05.309929 11228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:05.311249 11134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } }
I20260929 03:15:05.311517 11134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077f9da7e2a148f2a16d2741d7313d9d, State: Initialized, Role: FOLLOWER
Sep 29 03:15:05 dist-test-slave-k1dg krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
I20260929 03:15:05.311926 11134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } }
I20260929 03:15:05.312498 11230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } } }
I20260929 03:15:05.312722 11230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:05.313540 11134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:05.315897 11209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:05.316430 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.302596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.20.174.125:45763)
0929 03:15:05.303050 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.303054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.303069 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:15:05.303109 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.303111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.303156 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.303240 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.304159 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.304929 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.308635 (+ 3706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.308969 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.309051 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.312526 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.312530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.312533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.312562 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.314320 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.314892 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.314896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.314898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.314988 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.315300 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.315303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.315305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.315533 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.315700 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.316095 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.316236 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374}
I20260929 03:15:05.316897 11209 log.cc:824] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:05.319733 11209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4: No bootstrap required, opened a new log
I20260929 03:15:05.320582 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:15:05.322000 11032 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bde451a318c14fd481e1edad0ac015d8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37132
I20260929 03:15:05.322242 11032 raft_consensus.cc:482] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:15:05.322389 11032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:05.323751 11032 raft_consensus.cc:504] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } }
I20260929 03:15:05.323889 11209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } }
I20260929 03:15:05.324165 11209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55088875ed9f4f69bc81cf763a23e3e4, State: Initialized, Role: FOLLOWER
I20260929 03:15:05.324468 11209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } }
I20260929 03:15:05.324776 11032 leader_election.cc:288] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [CANDIDATE]: Term 1 election: Requested vote from peers 077f9da7e2a148f2a16d2741d7313d9d (127.20.174.125:45763), 55088875ed9f4f69bc81cf763a23e3e4 (127.20.174.124:33591)
I20260929 03:15:05.325248 11108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bde451a318c14fd481e1edad0ac015d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "077f9da7e2a148f2a16d2741d7313d9d"
I20260929 03:15:05.325495 11108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:05.325940 11209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:05.327026 11243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } } }
I20260929 03:15:05.327229 11243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:15:05.331761 11254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:05.331931 11254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:05.327358 11066 master_runner.cc:186] Error getting master registration for 127.20.174.124:33591: OK, instance_id { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" instance_seqno: 1790651705250536 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:15:05.327055 11179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bde451a318c14fd481e1edad0ac015d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "55088875ed9f4f69bc81cf763a23e3e4"
I20260929 03:15:05.333256 11179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:05.334821 11179 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde451a318c14fd481e1edad0ac015d8 in term 1.
I20260929 03:15:05.335251 11008 leader_election.cc:302] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55088875ed9f4f69bc81cf763a23e3e4, bde451a318c14fd481e1edad0ac015d8; no voters:
I20260929 03:15:05.335625 11216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:05.335875 11216 raft_consensus.cc:686] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [term 1 LEADER]: Becoming Leader. State: Replica: bde451a318c14fd481e1edad0ac015d8, State: Running, Role: LEADER
I20260929 03:15:05.335705 11108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bde451a318c14fd481e1edad0ac015d8 in term 1.
I20260929 03:15:05.336244 11216 consensus_queue.cc:237] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } }
I20260929 03:15:05.338251 11216 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde451a318c14fd481e1edad0ac015d8. Latest consensus state: current_term: 1 leader_uuid: "bde451a318c14fd481e1edad0ac015d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } } }
I20260929 03:15:05.338390 11216 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:05.335911 11066 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 03:15:05.339569 11242 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 03:15:05.339861 11140 master_runner.cc:263] No leader master found from master 55088875ed9f4f69bc81cf763a23e3e4
I20260929 03:15:05.339958 11140 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 03:15:05.339922 11242 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:05.341830 11263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:05.342927 11263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:05.349010 11179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [term 1 FOLLOWER]: Refusing update from remote peer bde451a318c14fd481e1edad0ac015d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:05.350188 11108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [term 1 FOLLOWER]: Refusing update from remote peer bde451a318c14fd481e1edad0ac015d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:05.351213 11216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:05.351890 11216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:05.361321 11243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde451a318c14fd481e1edad0ac015d8. Latest consensus state: current_term: 1 leader_uuid: "bde451a318c14fd481e1edad0ac015d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } } }
I20260929 03:15:05.361485 11243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:05.362814 11230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde451a318c14fd481e1edad0ac015d8. Latest consensus state: current_term: 1 leader_uuid: "bde451a318c14fd481e1edad0ac015d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } } }
I20260929 03:15:05.362965 11230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:05.374043 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.341257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.20.174.124:33591)
0929 03:15:05.341373 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.341377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.358809 (+ 17432us) server_negotiation.cc:407] Connection header received
0929 03:15:05.360286 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.360290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.360351 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.360407 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.361475 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.362279 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.363755 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.364128 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.364205 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.367228 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.367232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.367235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.367265 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.369011 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.371731 (+ 2720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.371735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.371737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.371841 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.372470 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.372473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.372475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.372697 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.372825 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.373696 (+ 871us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.373851 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:15:05.375929 11267 mvcc.cc:205] Tried to move back new op lower bound from 7334509385101557760 to 7334509385061519360. Current Snapshot: MvccSnapshot[applied={T|T < 7334509385101557760}]
I20260929 03:15:05.376801 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.340419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.20.174.125:45763)
0929 03:15:05.340546 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.340550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.348532 (+ 7982us) server_negotiation.cc:407] Connection header received
0929 03:15:05.349044 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.349048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.349105 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.349162 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.350616 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.351433 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.359070 (+ 7637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.359468 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.359562 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.369590 (+ 10028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.369594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.369598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.369630 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.371360 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.373179 (+ 1819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.373185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.373190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.373281 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.375609 (+ 2328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.375613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.375615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.375891 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.376032 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.376472 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.376616 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:15:05.376739 11216 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bde451a318c14fd481e1edad0ac015d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } } }
I20260929 03:15:05.376932 11216 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:05.379061 11230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bde451a318c14fd481e1edad0ac015d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } } }
I20260929 03:15:05.379191 11230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:05.381584 11269 mvcc.cc:205] Tried to move back new op lower bound from 7334509385101557760 to 7334509385061519360. Current Snapshot: MvccSnapshot[applied={T|T < 7334509385101557760}]
I20260929 03:15:05.385424 11216 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bde451a318c14fd481e1edad0ac015d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } } }
I20260929 03:15:05.385586 11216 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:05.387094 11243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bde451a318c14fd481e1edad0ac015d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bde451a318c14fd481e1edad0ac015d8" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 42123 } } peers { permanent_uuid: "077f9da7e2a148f2a16d2741d7313d9d" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 45763 } } peers { permanent_uuid: "55088875ed9f4f69bc81cf763a23e3e4" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 33591 } } }
I20260929 03:15:05.387233 11243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:05.388623 11268 mvcc.cc:205] Tried to move back new op lower bound from 7334509385101557760 to 7334509385061519360. Current Snapshot: MvccSnapshot[applied={T|T < 7334509385101557760}]
I20260929 03:15:05.389874 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:05.342071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.20.174.126:42123)
0929 03:15:05.342187 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.342191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.364951 (+ 22760us) server_negotiation.cc:407] Connection header received
0929 03:15:05.365073 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.365076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.365142 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.365241 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.366154 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.367081 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.369919 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:05.370346 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:05.370470 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:05.381840 (+ 11370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:05.381845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:05.381848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:05.381882 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:05.384013 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.388103 (+ 4090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.388107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.388109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.388222 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:05.388822 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:05.388826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:05.388828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:05.389074 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:05.389182 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:05.389577 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 03:15:05.389712 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:15:05.399159 11263 catalog_manager.cc:1347] Generated new cluster ID: 0d193c7ba0a3430a8118dcf9df06d491
I20260929 03:15:05.399257 11263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:05.433511 11263 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:05.435344 11263 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:05.462597 11263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bde451a318c14fd481e1edad0ac015d8: Generated new TSK 0
I20260929 03:15:05.466132 11263 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:15:05.503537 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.499738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.20.174.125:45763)
0929 03:15:05.499871 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.499876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.502101 (+ 2225us) server_negotiation.cc:407] Connection header received
0929 03:15:05.502324 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.502328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.502387 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.502496 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.503388 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39386: BlockingRecv error: recv got EOF from 127.0.0.1:39386 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 03:15:05.505052 11063 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.500701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.20.174.126:42123)
0929 03:15:05.500822 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.500828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.504065 (+ 3237us) server_negotiation.cc:407] Connection header received
0929 03:15:05.504247 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.504251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.504323 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.504377 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.504914 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37160: BlockingRecv error: recv got EOF from 127.0.0.1:37160 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 03:15:05.506659 11212 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.501166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.20.174.124:33591)
0929 03:15:05.501281 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.501287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.505493 (+ 4206us) server_negotiation.cc:407] Connection header received
0929 03:15:05.505779 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.505783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.505836 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.505889 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.506531 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55868: BlockingRecv error: recv got EOF from 127.0.0.1:55868 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 03:15:05.511773 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.509376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.20.174.125:45763)
0929 03:15:05.509502 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.509506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.510750 (+ 1244us) server_negotiation.cc:407] Connection header received
0929 03:15:05.510983 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.510987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.511067 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.511121 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.511662 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39402: BlockingRecv error: recv got EOF from 127.0.0.1:39402 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 03:15:05.519676 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.515665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.20.174.125:45763)
0929 03:15:05.515783 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.515788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.516346 (+ 558us) server_negotiation.cc:407] Connection header received
0929 03:15:05.516557 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.516560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.516617 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.516669 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.519513 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39412: BlockingRecv error: recv got EOF from 127.0.0.1:39412 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 03:15:05.527669 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.525749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.20.174.125:45763)
0929 03:15:05.525880 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.525884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.526573 (+ 689us) server_negotiation.cc:407] Connection header received
0929 03:15:05.526801 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.526805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.526862 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.526920 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.527519 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39420: BlockingRecv error: recv got EOF from 127.0.0.1:39420 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 03:15:05.537058 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.534981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.20.174.125:45763)
0929 03:15:05.535164 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.535169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.535955 (+ 786us) server_negotiation.cc:407] Connection header received
0929 03:15:05.536174 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.536178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.536235 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.536305 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.536931 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39432: BlockingRecv error: recv got EOF from 127.0.0.1:39432 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 03:15:05.549952 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.548017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.20.174.125:45763)
0929 03:15:05.548143 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.548147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.548881 (+ 734us) server_negotiation.cc:407] Connection header received
0929 03:15:05.549098 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.549102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.549159 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.549235 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.549818 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39442: BlockingRecv error: recv got EOF from 127.0.0.1:39442 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 03:15:05.569613 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.568412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.20.174.125:45763)
0929 03:15:05.568588 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.568592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.568641 (+ 49us) server_negotiation.cc:407] Connection header received
0929 03:15:05.568789 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.568792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.568847 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.568929 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.569452 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39450: BlockingRecv error: recv got EOF from 127.0.0.1:39450 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 03:15:05.605376 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.604084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.20.174.125:45763)
0929 03:15:05.604250 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.604254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.604398 (+ 144us) server_negotiation.cc:407] Connection header received
0929 03:15:05.604573 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.604577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.604643 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.604753 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.605266 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39456: BlockingRecv error: recv got EOF from 127.0.0.1:39456 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 03:15:05.672976 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.671646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.20.174.125:45763)
0929 03:15:05.671848 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.671851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.671954 (+ 103us) server_negotiation.cc:407] Connection header received
0929 03:15:05.672083 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.672085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.672143 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.672230 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.672780 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39464: BlockingRecv error: recv got EOF from 127.0.0.1:39464 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260929 03:15:05.807001 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:05.805568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.20.174.125:45763)
0929 03:15:05.805732 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:05.805739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:05.805757 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:15:05.805843 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:05.805847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:05.805915 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:05.806019 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:05.806818 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39470: BlockingRecv error: recv got EOF from 127.0.0.1:39470 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 03:15:06.067351 11137 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:06.066175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.20.174.125:45763)
0929 03:15:06.066342 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:06.066346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:06.066363 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:15:06.066581 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:06.066583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:06.066634 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:06.066711 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:06.067226 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39476: BlockingRecv error: recv got EOF from 127.0.0.1:39476 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:15:06.333343 11254 catalog_manager.cc:1259] Loaded cluster ID: 0d193c7ba0a3430a8118dcf9df06d491
I20260929 03:15:06.333433 11254 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4: loading cluster ID for follower catalog manager: success
I20260929 03:15:06.335495 11254 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4: acquiring CA information for follower catalog manager: success
I20260929 03:15:06.336712 11254 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55088875ed9f4f69bc81cf763a23e3e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:15:06.341261 11242 catalog_manager.cc:1259] Loaded cluster ID: 0d193c7ba0a3430a8118dcf9df06d491
I20260929 03:15:06.341341 11242 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d: loading cluster ID for follower catalog manager: success
I20260929 03:15:06.343180 11242 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d: acquiring CA information for follower catalog manager: success
I20260929 03:15:06.344406 11242 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 077f9da7e2a148f2a16d2741d7313d9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:15:06.585973 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:06.581754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.20.174.125:45763)
0929 03:15:06.582041 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:06.582049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:06.582120 (+ 71us) server_negotiation.cc:407] Connection header received
0929 03:15:06.582329 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:06.582333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:06.582513 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:06.582648 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:06.582911 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:06.583463 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:06.584388 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:06.584629 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:06.584758 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:06.585095 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:06.585539 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:06.585688 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:06.585734 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 03:15:06.585797 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260929 03:15:06.593222 11346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:06.589509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.20.174.124:33591)
0929 03:15:06.589755 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:06.589759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:06.589773 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:15:06.589858 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:06.589860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:06.589975 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:06.590079 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:06.590286 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:06.590910 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:06.591753 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:06.591975 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:06.592052 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:06.592321 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:06.592773 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:06.592871 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:06.592946 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 03:15:06.592999 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260929 03:15:06.601313 11346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:06.596815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.20.174.124:33591)
0929 03:15:06.596938 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:06.596943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:06.597152 (+ 209us) server_negotiation.cc:407] Connection header received
0929 03:15:06.597307 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:06.597310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:06.597387 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:06.597456 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:06.597865 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:06.598594 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:06.600107 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:06.600435 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:06.600547 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:06.600831 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:06.600911 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:06.600989 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:06.601083 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 03:15:06.601137 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:15:06.601465 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:06.596568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.20.174.125:45763)
0929 03:15:06.596704 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:06.596709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:06.597009 (+ 300us) server_negotiation.cc:407] Connection header received
0929 03:15:06.597231 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:06.597235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:06.597289 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:06.597376 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:06.597705 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:06.598610 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:06.600139 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:06.600513 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:06.600615 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:06.600932 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:06.601013 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:06.601060 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:06.601242 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 03:15:06.601312 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:15:06.601687 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:06.596800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.20.174.126:42123)
0929 03:15:06.597092 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:06.597097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:06.597109 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:06.597210 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:06.597213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:06.597416 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:06.597539 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:06.597848 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:06.598659 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:06.599868 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:06.600218 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:06.600330 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:06.600686 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:06.601093 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:06.601177 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:06.601468 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 03:15:06.601538 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260929 03:15:06.605583 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 10991
I20260929 03:15:06.612389 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 11066
I20260929 03:15:06.618731 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 11140
2026-09-29T03:15:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3285 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:06 dist-test-slave-k1dg krb5kdc[11357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:06 dist-test-slave-k1dg krb5kdc[11357](info): set up 2 sockets
Sep 29 03:15:06 dist-test-slave-k1dg krb5kdc[11357](info): commencing operation
krb5kdc: starting...
W20260929 03:15:07.743063 21177 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/test-user.keytab.
Sep 29 03:15:07 dist-test-slave-k1dg krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.174.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
Entry for principal kudu/127.20.174.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab.
I20260929 03:15:07.870800 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:36271
--webserver_interface=127.20.174.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:36271,127.20.174.125:38161,127.20.174.124:39087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:07.987520 11370 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:15:07.987869 11370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:07.987895 11370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:07.987982 11370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:07.992512 11370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:07.992594 11370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:07.992614 11370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:07.992631 11370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:07.992647 11370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:07.998616 11370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:36271,127.20.174.125:38161,127.20.174.124:39087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.126.keytab
--principal=kudu/127.20.174.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.126:36271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.174.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:07.999881 11370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:08.001047 11370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:08.010221 11376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:08.010341 11378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:08.010221 11375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:08.010653 11370 server_base.cc:1034] running on GCE node
I20260929 03:15:08.011344 11370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:08.012585 11370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:08.013842 11370 hybrid_clock.cc:648] HybridClock initialized: now 1790651708013808 us; error 49 us; skew 500 ppm
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:08.017033 11370 init.cc:377] Logged in from keytab as kudu/127.20.174.126@KRBTEST.COM (short username kudu)
I20260929 03:15:08.018069 11370 webserver.cc:460] Webserver started at http://127.20.174.126:32865/ using document root <none> and password file <none>
I20260929 03:15:08.018755 11370 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:08.018853 11370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:08.019111 11370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:08.021343 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/data/instance:
uuid: "e8a6cfb9b0954cab8420f63e36837a4e"
format_stamp: "Formatted at 2026-09-29 03:15:08 on dist-test-slave-k1dg"
server_key: "216443b7e33174d644181636460c3b5a"
server_key_iv: "f6cd31b8e63ec05377be561689971d47"
server_key_version: "encryptionkey@0"
I20260929 03:15:08.021981 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/wal/instance:
uuid: "e8a6cfb9b0954cab8420f63e36837a4e"
format_stamp: "Formatted at 2026-09-29 03:15:08 on dist-test-slave-k1dg"
server_key: "216443b7e33174d644181636460c3b5a"
server_key_iv: "f6cd31b8e63ec05377be561689971d47"
server_key_version: "encryptionkey@0"
I20260929 03:15:08.026252 11370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:15:08.029181 11385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:08.030689 11370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:15:08.030844 11370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/wal
uuid: "e8a6cfb9b0954cab8420f63e36837a4e"
format_stamp: "Formatted at 2026-09-29 03:15:08 on dist-test-slave-k1dg"
server_key: "216443b7e33174d644181636460c3b5a"
server_key_iv: "f6cd31b8e63ec05377be561689971d47"
server_key_version: "encryptionkey@0"
I20260929 03:15:08.030987 11370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:08.051139 11370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:08.054841 11370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:08.055073 11370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:08.062568 11370 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.126:36271
I20260929 03:15:08.062937 11437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.126:36271 every 8 connection(s)
I20260929 03:15:08.063660 11370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:08.066658 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:08.068169 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 11370
I20260929 03:15:08.068324 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-0/wal/instance
I20260929 03:15:08.068653 21177 external_mini_cluster.cc:1428] Setting key 0b4e699dc91b5efc6e323c1c6c261170
I20260929 03:15:08.071674 11438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.174.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.125@KRBTEST.COM" created.
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:15:08.090703 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.074303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:36271 (local address 127.0.0.1:60252)
0929 03:15:08.075066 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.075124 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:08.076422 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:08.077312 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:08.077318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:08.077822 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:08.078089 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.078105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.079568 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.079572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.080645 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.080667 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.081682 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.081685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.081887 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.082589 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:08.082607 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:08.084491 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:08.087299 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.087308 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.087318 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.087596 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.088033 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.088035 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.088037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.088154 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.088998 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:08.089004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:08.089152 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 03:15:08.089311 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 03:15:08.089605 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":73,"thread_start_us":246,"threads_started":1}
I20260929 03:15:08.090703 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.074303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.20.174.126:36271)
0929 03:15:08.074854 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.074874 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.075273 (+ 399us) server_negotiation.cc:407] Connection header received
0929 03:15:08.076831 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.076837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.077063 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.077320 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.078300 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.079414 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.080783 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.081487 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.081741 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.084676 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.084686 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.084705 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.084738 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.087146 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.087741 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.087749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.087752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.087924 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.088258 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.088261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.088263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.088854 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.089023 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.089439 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.089712 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":376,"thread_start_us":196,"threads_started":1}
I20260929 03:15:08.093449 11438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:08.094302 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.094024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:38161 (local address 127.0.0.1:40342)
0929 03:15:08.094156 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.094232 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 03:15:08.094559 11386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.174.125:38161: connect: Connection refused (error 111)
I20260929 03:15:08.095553 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.095302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:38161 (local address 127.0.0.1:40358)
0929 03:15:08.095417 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.095464 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 03:15:08.095762 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.125:38161: Network error: Client connection negotiation failed: client connection to 127.20.174.125:38161: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab.
Entry for principal kudu/127.20.174.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab.
I20260929 03:15:08.102566 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.125:38161
--webserver_interface=127.20.174.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab
--principal=kudu/127.20.174.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:36271,127.20.174.125:38161,127.20.174.124:39087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:15:08.145114 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } attempt: 1
I20260929 03:15:08.146076 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.145750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:38161 (local address 127.0.0.1:40360)
0929 03:15:08.145918 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.145995 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 03:15:08.146970 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.146710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:38161 (local address 127.0.0.1:40372)
0929 03:15:08.146830 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.146882 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 03:15:08.147136 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.125:38161: Network error: Client connection negotiation failed: client connection to 127.20.174.125:38161: connect: Connection refused (error 111)
I20260929 03:15:08.215322 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } attempt: 2
I20260929 03:15:08.216306 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.215980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:38161 (local address 127.0.0.1:40378)
0929 03:15:08.216151 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.216225 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 03:15:08.217216 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.216928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:38161 (local address 127.0.0.1:40380)
0929 03:15:08.217078 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.217141 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.125:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 03:15:08.217374 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.125:38161: Network error: Client connection negotiation failed: client connection to 127.20.174.125:38161: connect: Connection refused (error 111)
W20260929 03:15:08.222081 11445 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:15:08.222415 11445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:08.222523 11445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:08.222626 11445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:08.227909 11445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:08.227985 11445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:08.228005 11445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:08.228024 11445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:08.228039 11445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:08.234174 11445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:36271,127.20.174.125:38161,127.20.174.124:39087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.125.keytab
--principal=kudu/127.20.174.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.125:38161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.174.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:08.235674 11445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:08.236802 11445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:15:08.246289 11445 server_base.cc:1034] running on GCE node
W20260929 03:15:08.246222 11450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:08.246222 11453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:08.246217 11451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:08.247180 11445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:08.248415 11445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:08.249590 11445 hybrid_clock.cc:648] HybridClock initialized: now 1790651708249570 us; error 35 us; skew 500 ppm
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:08.252650 11445 init.cc:377] Logged in from keytab as kudu/127.20.174.125@KRBTEST.COM (short username kudu)
I20260929 03:15:08.253638 11445 webserver.cc:460] Webserver started at http://127.20.174.125:46541/ using document root <none> and password file <none>
I20260929 03:15:08.254231 11445 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:08.254326 11445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:08.254627 11445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:08.256785 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/data/instance:
uuid: "c331a0fd376b482f8edca6fab6d41d8c"
format_stamp: "Formatted at 2026-09-29 03:15:08 on dist-test-slave-k1dg"
server_key: "cb07e9706659ff16bee749e09357a178"
server_key_iv: "f3a6bde798b41718b6dce1078488f692"
server_key_version: "encryptionkey@0"
I20260929 03:15:08.257375 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/wal/instance:
uuid: "c331a0fd376b482f8edca6fab6d41d8c"
format_stamp: "Formatted at 2026-09-29 03:15:08 on dist-test-slave-k1dg"
server_key: "cb07e9706659ff16bee749e09357a178"
server_key_iv: "f3a6bde798b41718b6dce1078488f692"
server_key_version: "encryptionkey@0"
I20260929 03:15:08.261790 11445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:15:08.264560 11460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:08.265976 11445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:15:08.266148 11445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/wal
uuid: "c331a0fd376b482f8edca6fab6d41d8c"
format_stamp: "Formatted at 2026-09-29 03:15:08 on dist-test-slave-k1dg"
server_key: "cb07e9706659ff16bee749e09357a178"
server_key_iv: "f3a6bde798b41718b6dce1078488f692"
server_key_version: "encryptionkey@0"
I20260929 03:15:08.266283 11445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:08.285399 11445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:08.289191 11445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:08.289551 11445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:08.297503 11445 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.125:38161
I20260929 03:15:08.297806 11512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.125:38161 every 8 connection(s)
I20260929 03:15:08.298625 11445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/data/info.pb
I20260929 03:15:08.299979 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 11445
I20260929 03:15:08.300168 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-1/wal/instance
I20260929 03:15:08.300518 21177 external_mini_cluster.cc:1428] Setting key e12dc35a4c73d53c94cd63cab97d8b52
I20260929 03:15:08.302460 11513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:08.306684 11513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.174.124@KRBTEST.COM; defaulting to no policy
I20260929 03:15:08.308552 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.174.124@KRBTEST.COM" created.
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:15:08.322741 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.309097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:38161 (local address 127.0.0.1:40396)
0929 03:15:08.309209 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.309225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:08.309321 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:08.311452 (+ 2131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:08.311454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:08.311753 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:08.311859 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.311866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.313840 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.313845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.314366 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.314374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.315240 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.315247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.315397 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.315985 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:08.316006 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:08.317652 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:08.320127 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.320130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.320133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.320395 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.320856 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.320860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.320862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.320963 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.321921 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:08.321924 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:08.322046 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 03:15:08.322225 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 03:15:08.322395 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 03:15:08.323971 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.309295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.20.174.125:38161)
0929 03:15:08.309834 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.309856 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.309900 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:15:08.310925 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.310951 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.311261 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.311459 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.312041 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.313641 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.314555 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.315086 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.315269 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.317819 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.317831 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.317843 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.317865 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.320023 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.320543 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.320553 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.320564 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.320725 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.321088 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.321091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.321093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.321768 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.321916 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.322411 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.322909 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":350,"thread_start_us":140,"threads_started":1}
I20260929 03:15:08.324067 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.308826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.20.174.126:36271)
0929 03:15:08.308989 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.308997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.310201 (+ 1204us) server_negotiation.cc:407] Connection header received
0929 03:15:08.311145 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.311148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.311228 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.311303 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.312724 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.313254 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.314807 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.315731 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.315847 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.320020 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.320026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.320029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.320065 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.321769 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.322382 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.322385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.322387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.322501 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.323090 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.323094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.323096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.323286 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.323395 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.323786 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.323907 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:15:08.324113 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.309295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:36271 (local address 127.0.0.1:60258)
0929 03:15:08.310018 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.310067 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:08.310846 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:08.311386 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:08.311398 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:08.312378 (+ 980us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:08.312575 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.312585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.313374 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.313379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.314669 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.314684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.316741 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.316753 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.316929 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.317539 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:08.317567 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:08.319840 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:08.321921 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.321931 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.321937 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.322252 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.322596 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.322599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.322601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.322794 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.323394 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:08.323402 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:08.323524 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 03:15:08.323684 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 03:15:08.323906 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":3,"spinlock_wait_cycles":29952,"thread_start_us":169,"threads_started":1}
I20260929 03:15:08.325373 11513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:08.326267 11438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:08.327034 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.326788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56292)
0929 03:15:08.326905 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.326963 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:15:08.327872 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.327575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56300)
0929 03:15:08.327721 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.327763 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 03:15:08.328061 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:39087: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.174.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab.
Entry for principal kudu/127.20.174.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab.
I20260929 03:15:08.330071 21177 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEa40KE/build/debug/bin/kudu
/tmp/dist-test-taskEa40KE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.124:39087
--webserver_interface=127.20.174.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab
--principal=kudu/127.20.174.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.174.84:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.174.126:36271,127.20.174.125:38161,127.20.174.124:39087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
I20260929 03:15:08.334898 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.325889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:38161 (local address 127.0.0.1:40406)
0929 03:15:08.326274 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.326286 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:08.326356 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:08.326777 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:08.326783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:08.326956 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:08.327081 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.327088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.328007 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.328010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.328593 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.328601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.329066 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.329070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.329168 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.329679 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:08.329692 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:08.331183 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:08.333241 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.333248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.333251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.333448 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.333732 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.333735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.333736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.333819 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.334222 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:08.334225 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:08.334335 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 03:15:08.334464 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 03:15:08.334664 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":1}
I20260929 03:15:08.334898 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.325944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.20.174.125:38161)
0929 03:15:08.326055 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.326059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.326352 (+ 293us) server_negotiation.cc:407] Connection header received
0929 03:15:08.326503 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.326506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.326551 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.326639 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.327207 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.327896 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.328716 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.328959 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.329067 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.331331 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.331335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.331337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.331365 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.333126 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.333546 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.333551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.333553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.333634 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.333921 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.333925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.333927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.334118 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.334223 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.334502 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.334714 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":41}
I20260929 03:15:08.335780 11513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:08.336517 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.336242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56310)
0929 03:15:08.336359 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.336456 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 03:15:08.337086 11462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
I20260929 03:15:08.338104 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.337875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56322)
0929 03:15:08.337991 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.338038 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 03:15:08.338277 11513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:39087: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
I20260929 03:15:08.359234 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } attempt: 1
I20260929 03:15:08.360203 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.359843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56330)
0929 03:15:08.360057 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.360120 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 03:15:08.360937 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.360713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56344)
0929 03:15:08.360822 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.360874 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 03:15:08.361064 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:39087: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
I20260929 03:15:08.387607 11513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } attempt: 1
I20260929 03:15:08.388569 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.388227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56348)
0929 03:15:08.388415 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.388484 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 03:15:08.389446 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.389196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56352)
0929 03:15:08.389331 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.389382 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 03:15:08.389601 11513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:39087: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
I20260929 03:15:08.436223 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } attempt: 2
I20260929 03:15:08.437103 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.436756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56364)
0929 03:15:08.436958 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.437017 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 03:15:08.437831 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.437620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56372)
0929 03:15:08.437725 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.437778 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 03:15:08.437968 11438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:39087: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
W20260929 03:15:08.453462 11519 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:15:08.453899 11519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:08.453970 11519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:08.454053 11519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 03:15:08.457753 11513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } attempt: 2
W20260929 03:15:08.458688 11519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260929 03:15:08.458701 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.458349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56378)
0929 03:15:08.458556 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.458621 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 03:15:08.458773 11519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:08.458813 11519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:08.458851 11519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:08.458873 11519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:08.459471 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.459241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56394)
0929 03:15:08.459358 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.459414 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 03:15:08.459602 11513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.174.124:39087: Network error: Client connection negotiation failed: client connection to 127.20.174.124:39087: connect: Connection refused (error 111)
I20260929 03:15:08.464752 11519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.174.84:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.174.126:36271,127.20.174.125:38161,127.20.174.124:39087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/krb5kdc/kudu_127.20.174.124.keytab
--principal=kudu/127.20.174.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.174.124:39087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.174.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:08.466001 11519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:08.467278 11519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:08.476390 11526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:08.476392 11528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:08.476418 11525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:08.476706 11519 server_base.cc:1034] running on GCE node
I20260929 03:15:08.477587 11519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:08.478909 11519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:08.480110 11519 hybrid_clock.cc:648] HybridClock initialized: now 1790651708480082 us; error 47 us; skew 500 ppm
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:08.483409 11519 init.cc:377] Logged in from keytab as kudu/127.20.174.124@KRBTEST.COM (short username kudu)
I20260929 03:15:08.484417 11519 webserver.cc:460] Webserver started at http://127.20.174.124:46305/ using document root <none> and password file <none>
I20260929 03:15:08.485059 11519 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:08.485167 11519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:08.485420 11519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:08.487805 11519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/data/instance:
uuid: "b18951bb81824b3da21010773cb7bc44"
format_stamp: "Formatted at 2026-09-29 03:15:08 on dist-test-slave-k1dg"
server_key: "ce68edce9f27192c13d73e509e44a2f6"
server_key_iv: "b82cd8a159c2f46b66882cbbfd42a1f0"
server_key_version: "encryptionkey@0"
I20260929 03:15:08.488445 11519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/wal/instance:
uuid: "b18951bb81824b3da21010773cb7bc44"
format_stamp: "Formatted at 2026-09-29 03:15:08 on dist-test-slave-k1dg"
server_key: "ce68edce9f27192c13d73e509e44a2f6"
server_key_iv: "b82cd8a159c2f46b66882cbbfd42a1f0"
server_key_version: "encryptionkey@0"
I20260929 03:15:08.492942 11519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:15:08.495724 11535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:08.497180 11519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:15:08.497408 11519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/data,/tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/wal
uuid: "b18951bb81824b3da21010773cb7bc44"
format_stamp: "Formatted at 2026-09-29 03:15:08 on dist-test-slave-k1dg"
server_key: "ce68edce9f27192c13d73e509e44a2f6"
server_key_iv: "b82cd8a159c2f46b66882cbbfd42a1f0"
server_key_version: "encryptionkey@0"
I20260929 03:15:08.497553 11519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:08.511462 11519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:08.514876 11519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:08.515108 11519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:08.522238 11519 rpc_server.cc:307] RPC server started. Bound to: 127.20.174.124:39087
I20260929 03:15:08.522547 11587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.174.124:39087 every 8 connection(s)
I20260929 03:15:08.523357 11519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/data/info.pb
I20260929 03:15:08.526190 11588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:08.527949 21177 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEa40KE/build/debug/bin/kudu as pid 11519
I20260929 03:15:08.528090 21177 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEa40KE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651413504048-21177-0/minicluster-data/master-2/wal/instance
I20260929 03:15:08.528443 21177 external_mini_cluster.cc:1428] Setting key e442c7e4b50d330639fd147ab46e88dc
I20260929 03:15:08.531538 11588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:08.537145 11438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } attempt: 3
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.126@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
I20260929 03:15:08.545145 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.533736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.20.174.126:36271)
0929 03:15:08.533889 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.533894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.534234 (+ 340us) server_negotiation.cc:407] Connection header received
0929 03:15:08.534416 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.534420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.534518 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.534603 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.535475 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.536160 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.537200 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.537547 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.537667 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.540767 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.540772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.540775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.540803 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.542855 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.543413 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.543416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.543418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.543511 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.543864 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.543867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.543869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.544121 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.544224 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.544795 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.544925 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":896}
I20260929 03:15:08.546048 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.533685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.126:36271 (local address 127.0.0.1:60262)
0929 03:15:08.534082 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.534151 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:08.534288 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:08.534651 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:08.534656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:08.535187 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:08.535348 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.535356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.536279 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.536282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.537093 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.537104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.537836 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.537842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.537957 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.538886 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:08.538911 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:08.540610 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:08.542990 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.542999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.543006 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.543307 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.543620 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.543622 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.543624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.543753 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.544286 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:08.544294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:08.544497 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 03:15:08.545306 (+ 809us) client_negotiation.cc:240] Negotiation successful
0929 03:15:08.545566 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":145,"threads_started":1}
I20260929 03:15:08.546046 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.529939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.20.174.124:39087)
0929 03:15:08.530630 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.530650 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.530693 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:15:08.531679 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.531706 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.532191 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.532624 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.533482 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.534854 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.535944 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.536579 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.536693 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.540042 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.540049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.540063 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.540094 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.542211 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.542783 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.542789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.542796 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.542918 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.543247 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.543250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.543252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.544067 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.544289 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.544561 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.544999 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"spinlock_wait_cycles":1408,"thread_start_us":132,"threads_started":1}
I20260929 03:15:08.547391 11588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:08.547780 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.537601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56404)
0929 03:15:08.537943 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.537958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:08.538039 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:08.538329 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:08.538334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:08.538556 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:08.538652 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.538661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.539853 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.539858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.540657 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.540670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.541313 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.541319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.541416 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.541839 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:08.541853 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:08.543452 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:08.546155 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.546158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.546161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.546418 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.546746 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.546749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.546751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.546844 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.547248 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:08.547254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:08.547364 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 03:15:08.547467 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 03:15:08.547607 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260929 03:15:08.549468 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.537666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.20.174.124:39087)
0929 03:15:08.537975 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.537995 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.538009 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:15:08.538132 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.538135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.538208 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.538296 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.538854 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.539660 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.540782 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.541171 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.541290 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.543587 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.543590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.543593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.543615 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.546033 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.546549 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.546552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.546554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.546645 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.546951 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.546954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.546956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.547140 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.549083 (+ 1943us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.549165 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.549288 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":136,"threads_started":1}
I20260929 03:15:08.551784 11513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } attempt: 3
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
I20260929 03:15:08.553637 11438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e: Bootstrap starting.
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.126@KRBTEST.COM
I20260929 03:15:08.556624 11438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:08.556823 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.548015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.20.174.125:38161)
0929 03:15:08.548183 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.548188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.548204 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:15:08.548248 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.548250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.548299 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.548392 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.548752 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.549506 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.550774 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.551088 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.551204 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.553485 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.553489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.553492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.553520 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.555150 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.555571 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.555577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.555580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.555664 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.556005 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.556011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.556016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.556213 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.556269 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.556562 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.556692 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 03:15:08.556820 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.547919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.125:38161 (local address 127.0.0.1:40422)
0929 03:15:08.548076 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.548090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:08.548165 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:08.548404 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:08.548406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:08.548535 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:08.548627 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.548632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.549864 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.549866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.550636 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.550649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.551214 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.551218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.551316 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.551889 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:08.551906 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:08.553323 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:08.555262 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.555267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.555269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.555470 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.555776 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.555781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.555784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.555888 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.556330 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:08.556332 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:08.556403 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 03:15:08.556510 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 03:15:08.556651 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 03:15:08.557623 11588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:15:08.557629 11438 log.cc:824] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:08.560389 11438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e: No bootstrap required, opened a new log
I20260929 03:15:08.560787 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.549541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.20.174.126:36271)
0929 03:15:08.549648 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.549652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.549763 (+ 111us) server_negotiation.cc:407] Connection header received
0929 03:15:08.549942 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.549945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.549992 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.550044 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.550892 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.551557 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.552736 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.553040 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.553153 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.556075 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.556078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.556081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.556108 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.558299 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.559210 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.559214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.559216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.559306 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.559640 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.559643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.559645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.559973 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.560141 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.560502 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.560623 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.124@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.174.125@KRBTEST.COM for kudu/127.20.174.124@KRBTEST.COM
I20260929 03:15:08.565709 11438 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } }
I20260929 03:15:08.566044 11438 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a6cfb9b0954cab8420f63e36837a4e, State: Initialized, Role: FOLLOWER
I20260929 03:15:08.566483 11438 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } }
I20260929 03:15:08.567054 11595 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } } }
I20260929 03:15:08.567214 11595 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:08.568259 11438 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:08.568665 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.558654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.20.174.124:39087)
0929 03:15:08.558902 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.558908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.558918 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:15:08.558944 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.558946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.558985 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.559077 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.559483 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.560225 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.561187 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.561564 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.561979 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.564627 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.564631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.564633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.564653 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.566487 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.567332 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.567336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.567338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.567429 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.567762 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.567765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.567767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.567980 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.568088 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.568384 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.568540 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":158}
I20260929 03:15:08.568665 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.558407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56422)
0929 03:15:08.558755 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.558771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:08.558839 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:08.559087 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:08.559089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:08.559264 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:08.559369 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.559378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.560341 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.560345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.561058 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.561068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.561674 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.561677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.561770 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.562236 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:08.562254 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:08.564506 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:08.566593 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.566596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.566598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.566865 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.567534 (+ 669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.567537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.567538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.567631 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.568101 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:08.568105 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:08.568206 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 03:15:08.568341 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 03:15:08.568486 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
Sep 29 03:15:08 dist-test-slave-k1dg krb5kdc[11357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.174.125@KRBTEST.COM
I20260929 03:15:08.570806 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.552185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.174.124:39087 (local address 127.0.0.1:56418)
0929 03:15:08.552435 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:08.552452 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:08.552529 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:08.558688 (+ 6159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:08.558691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:08.558895 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:08.559004 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.559011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.560180 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.560184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.561056 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:08.561070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.561829 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.561833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:08.561924 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.562388 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:08.562403 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:08.564525 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:08.568307 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.568310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.568312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.568583 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.569447 (+ 864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:08.569454 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:08.569460 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:08.569575 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:08.570112 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:08.570118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:08.570260 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 03:15:08.570374 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 03:15:08.570579 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260929 03:15:08.571789 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.558161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.20.174.124:39087)
0929 03:15:08.558345 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.558352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.558369 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:15:08.558409 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.558414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.558518 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.559128 (+ 610us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.559233 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.560008 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.561244 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.561681 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.562152 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.564694 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.564699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.564702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.564728 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.568177 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.569199 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.569204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.569207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.569306 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.569756 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.569762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.569766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.569984 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.571394 (+ 1410us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.571475 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.571616 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":72}
I20260929 03:15:08.572383 11588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44: Bootstrap starting.
I20260929 03:15:08.575488 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.562994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.20.174.125:38161)
0929 03:15:08.563133 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.563151 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.563223 (+ 72us) server_negotiation.cc:407] Connection header received
0929 03:15:08.563368 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.563372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.563417 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.563512 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.564358 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.566056 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.567208 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.567501 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.567561 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.571348 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.571352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.571355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.571378 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.573586 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.574085 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.574089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.574091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.574169 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.574489 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.574492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.574494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.574710 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.574880 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.575242 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.575352 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:15:08.575839 11513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c: Bootstrap starting.
W20260929 03:15:08.576889 11606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:08.577090 11606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:08.573989 11370 master_runner.cc:186] Error getting master registration for 127.20.174.125:38161: OK, instance_id { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" instance_seqno: 1790651708295342 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:15:08.578173 11370 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:15:08.579083 21177 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:15:08.579722 11588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:08.580317 11412 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e8a6cfb9b0954cab8420f63e36837a4e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60268
I20260929 03:15:08.580554 11412 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:15:08.580669 11412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:08.580659 11588 log.cc:824] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:08.580822 11513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:08.581765 11513 log.cc:824] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:08.582078 11412 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } }
I20260929 03:15:08.582871 11588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44: No bootstrap required, opened a new log
I20260929 03:15:08.583117 11412 leader_election.cc:288] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [CANDIDATE]: Term 1 election: Requested vote from peers c331a0fd376b482f8edca6fab6d41d8c (127.20.174.125:38161), b18951bb81824b3da21010773cb7bc44 (127.20.174.124:39087)
I20260929 03:15:08.584352 11513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c: No bootstrap required, opened a new log
I20260929 03:15:08.586611 11588 raft_consensus.cc:348] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } }
I20260929 03:15:08.586808 11588 raft_consensus.cc:729] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18951bb81824b3da21010773cb7bc44, State: Initialized, Role: FOLLOWER
I20260929 03:15:08.587059 11588 consensus_queue.cc:260] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } }
I20260929 03:15:08.587479 11614 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } } }
I20260929 03:15:08.587605 11614 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:08.588310 11588 sys_catalog.cc:564] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:08.588709 11562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b18951bb81824b3da21010773cb7bc44"
I20260929 03:15:08.588948 11562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:08.589105 11513 raft_consensus.cc:348] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } }
I20260929 03:15:08.589362 11513 raft_consensus.cc:729] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c331a0fd376b482f8edca6fab6d41d8c, State: Initialized, Role: FOLLOWER
I20260929 03:15:08.589708 11513 consensus_queue.cc:260] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } }
I20260929 03:15:08.590580 11562 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8a6cfb9b0954cab8420f63e36837a4e in term 1.
I20260929 03:15:08.591645 11513 sys_catalog.cc:564] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:08.591864 11487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c331a0fd376b482f8edca6fab6d41d8c"
I20260929 03:15:08.592108 11487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:08.593732 11487 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8a6cfb9b0954cab8420f63e36837a4e in term 1.
I20260929 03:15:08.594235 11386 leader_election.cc:302] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c331a0fd376b482f8edca6fab6d41d8c, e8a6cfb9b0954cab8420f63e36837a4e; no voters:
I20260929 03:15:08.594594 11595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:08.594887 11595 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [term 1 LEADER]: Becoming Leader. State: Replica: e8a6cfb9b0954cab8420f63e36837a4e, State: Running, Role: LEADER
I20260929 03:15:08.595309 11595 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } }
W20260929 03:15:08.597256 11628 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:08.597401 11628 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:08.597379 11595 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8a6cfb9b0954cab8420f63e36837a4e. Latest consensus state: current_term: 1 leader_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } } }
I20260929 03:15:08.597507 11595 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:08.599416 11519 master_runner.cc:263] No leader master found from master b18951bb81824b3da21010773cb7bc44
I20260929 03:15:08.599489 11519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:15:08.609692 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.590994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.20.174.126:36271)
0929 03:15:08.591143 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.591148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.591292 (+ 144us) server_negotiation.cc:407] Connection header received
0929 03:15:08.591524 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.591528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.591586 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.591649 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.597036 (+ 5387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.597749 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.602632 (+ 4883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.602996 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.603078 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.605509 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.605513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.605516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.605549 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.607448 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.608079 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.608084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.608086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.608184 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.608504 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.608508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.608510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.608783 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.608921 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.609365 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.609497 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:15:08.610644 11625 sys_catalog.cc:455] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } } }
I20260929 03:15:08.610859 11625 sys_catalog.cc:458] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:08.613360 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.591286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.20.174.124:39087)
0929 03:15:08.591392 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.591396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.597714 (+ 6318us) server_negotiation.cc:407] Connection header received
0929 03:15:08.597953 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.597956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.598013 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.598069 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.599290 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.599980 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.601102 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.601441 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.601518 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.605257 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.605262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.605265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.605306 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.607418 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.611345 (+ 3927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.611349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.611351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.611717 (+ 366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.612316 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.612320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.612322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.612557 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.612690 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.613063 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.613196 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 03:15:08.616700 11640 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:08.616853 11640 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:08.618777 11644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:08.619458 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:08.592293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.20.174.125:38161)
0929 03:15:08.592407 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.592411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.602454 (+ 10043us) server_negotiation.cc:407] Connection header received
0929 03:15:08.602628 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.602631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.602692 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.602739 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.607980 (+ 5241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.608766 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.609966 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:08.610323 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:08.610403 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:08.615649 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:08.615653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:08.615656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:08.615684 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:08.617409 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.617945 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.617948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.617950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.618041 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:08.618370 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:08.618373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:08.618375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:08.618632 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:08.618806 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:08.619156 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 03:15:08.619291 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:15:08.620432 11644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:08.625888 11487 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [term 1 FOLLOWER]: Refusing update from remote peer e8a6cfb9b0954cab8420f63e36837a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:08.625932 11562 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [term 1 FOLLOWER]: Refusing update from remote peer e8a6cfb9b0954cab8420f63e36837a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:08.626588 11641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:08.628455 11641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:08.631348 11625 sys_catalog.cc:455] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8a6cfb9b0954cab8420f63e36837a4e. Latest consensus state: current_term: 1 leader_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } } }
I20260929 03:15:08.631481 11625 sys_catalog.cc:458] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:08.632236 11614 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8a6cfb9b0954cab8420f63e36837a4e. Latest consensus state: current_term: 1 leader_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } } }
I20260929 03:15:08.632364 11614 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:08.636435 11641 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } } }
I20260929 03:15:08.636583 11641 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:08.637917 11625 sys_catalog.cc:455] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } } }
I20260929 03:15:08.638043 11625 sys_catalog.cc:458] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:08.638418 11646 mvcc.cc:205] Tried to move back new op lower bound from 7334509398524784640 to 7334509398410539008. Current Snapshot: MvccSnapshot[applied={T|T < 7334509398524784640}]
I20260929 03:15:08.640059 11647 mvcc.cc:205] Tried to move back new op lower bound from 7334509398524784640 to 7334509398410539008. Current Snapshot: MvccSnapshot[applied={T|T < 7334509398524784640}]
I20260929 03:15:08.647565 11644 catalog_manager.cc:1347] Generated new cluster ID: 1109a3cbf03e47c0a0268fe76a6f7237
I20260929 03:15:08.647675 11644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:08.661948 11595 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } } }
I20260929 03:15:08.662113 11595 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:08.663866 11614 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a6cfb9b0954cab8420f63e36837a4e" member_type: VOTER last_known_addr { host: "127.20.174.126" port: 36271 } } peers { permanent_uuid: "c331a0fd376b482f8edca6fab6d41d8c" member_type: VOTER last_known_addr { host: "127.20.174.125" port: 38161 } } peers { permanent_uuid: "b18951bb81824b3da21010773cb7bc44" member_type: VOTER last_known_addr { host: "127.20.174.124" port: 39087 } } }
I20260929 03:15:08.663990 11614 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:08.665633 11649 mvcc.cc:205] Tried to move back new op lower bound from 7334509398524784640 to 7334509398410539008. Current Snapshot: MvccSnapshot[applied={T|T < 7334509398524784640}]
I20260929 03:15:08.696434 11644 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:08.697754 11644 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:08.720746 11644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8a6cfb9b0954cab8420f63e36837a4e: Generated new TSK 0
I20260929 03:15:08.721752 11644 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:15:08.759008 11591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:08.755897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.20.174.124:39087)
0929 03:15:08.756027 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.756032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.757891 (+ 1859us) server_negotiation.cc:407] Connection header received
0929 03:15:08.758085 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.758088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.758140 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.758191 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.758870 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56446: BlockingRecv error: recv got EOF from 127.0.0.1:56446 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 03:15:08.760466 11441 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:08.756717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.20.174.126:36271)
0929 03:15:08.756845 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.756850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.759505 (+ 2655us) server_negotiation.cc:407] Connection header received
0929 03:15:08.759674 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.759678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.759732 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.759804 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.760346 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60294: BlockingRecv error: recv got EOF from 127.0.0.1:60294 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 03:15:08.761875 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:08.757341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.20.174.125:38161)
0929 03:15:08.757448 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.757452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.760973 (+ 3521us) server_negotiation.cc:407] Connection header received
0929 03:15:08.761136 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.761139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.761200 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.761252 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.761733 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40448: BlockingRecv error: recv got EOF from 127.0.0.1:40448 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 03:15:08.766549 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:08.765019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.20.174.125:38161)
0929 03:15:08.765125 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.765130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.765564 (+ 434us) server_negotiation.cc:407] Connection header received
0929 03:15:08.765790 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.765793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.765868 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.765922 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.766419 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40450: BlockingRecv error: recv got EOF from 127.0.0.1:40450 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 03:15:08.772028 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:08.770358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.20.174.125:38161)
0929 03:15:08.770513 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.770518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.771002 (+ 484us) server_negotiation.cc:407] Connection header received
0929 03:15:08.771206 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.771209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.771262 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.771315 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.771922 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40466: BlockingRecv error: recv got EOF from 127.0.0.1:40466 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 03:15:08.778064 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:08.776467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.20.174.125:38161)
0929 03:15:08.776574 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.776579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.777075 (+ 496us) server_negotiation.cc:407] Connection header received
0929 03:15:08.777263 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.777266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.777318 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.777374 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.777945 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40478: BlockingRecv error: recv got EOF from 127.0.0.1:40478 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 03:15:08.786692 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:08.784742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.20.174.125:38161)
0929 03:15:08.784917 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.784922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.785556 (+ 634us) server_negotiation.cc:407] Connection header received
0929 03:15:08.785788 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.785792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.785849 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.785905 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.786567 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40488: BlockingRecv error: recv got EOF from 127.0.0.1:40488 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260929 03:15:08.799543 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:08.797566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.20.174.125:38161)
0929 03:15:08.797697 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.797702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.798329 (+ 627us) server_negotiation.cc:407] Connection header received
0929 03:15:08.798635 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.798639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.798703 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.798760 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.799414 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40502: BlockingRecv error: recv got EOF from 127.0.0.1:40502 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 03:15:08.820051 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:08.818483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.20.174.125:38161)
0929 03:15:08.818615 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.818620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.818882 (+ 262us) server_negotiation.cc:407] Connection header received
0929 03:15:08.819082 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.819085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.819146 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.819246 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.819903 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40508: BlockingRecv error: recv got EOF from 127.0.0.1:40508 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 03:15:08.855587 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:08.854357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.20.174.125:38161)
0929 03:15:08.854519 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.854524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.854701 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:15:08.854860 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.854862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.854915 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.854993 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.855459 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40520: BlockingRecv error: recv got EOF from 127.0.0.1:40520 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 03:15:08.923308 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:08.921936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.20.174.125:38161)
0929 03:15:08.922124 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:08.922132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:08.922261 (+ 129us) server_negotiation.cc:407] Connection header received
0929 03:15:08.922390 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:08.922393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:08.922468 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:08.922544 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:08.923151 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40526: BlockingRecv error: recv got EOF from 127.0.0.1:40526 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 03:15:09.055100 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:09.053622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.20.174.125:38161)
0929 03:15:09.053801 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:09.053806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:09.054047 (+ 241us) server_negotiation.cc:407] Connection header received
0929 03:15:09.054262 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:09.054265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:09.054317 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:09.054418 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:09.054944 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40536: BlockingRecv error: recv got EOF from 127.0.0.1:40536 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 03:15:09.314704 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:09.313468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.20.174.125:38161)
0929 03:15:09.313631 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:09.313636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:09.313845 (+ 209us) server_negotiation.cc:407] Connection header received
0929 03:15:09.313915 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:09.313918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:09.313965 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:09.314059 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:09.314592 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40538: BlockingRecv error: recv got EOF from 127.0.0.1:40538 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:15:09.598780 11628 catalog_manager.cc:1259] Loaded cluster ID: 1109a3cbf03e47c0a0268fe76a6f7237
I20260929 03:15:09.598905 11628 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44: loading cluster ID for follower catalog manager: success
I20260929 03:15:09.600651 11628 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44: acquiring CA information for follower catalog manager: success
I20260929 03:15:09.602065 11628 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b18951bb81824b3da21010773cb7bc44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:15:09.618240 11640 catalog_manager.cc:1259] Loaded cluster ID: 1109a3cbf03e47c0a0268fe76a6f7237
I20260929 03:15:09.618326 11640 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c: loading cluster ID for follower catalog manager: success
I20260929 03:15:09.620152 11640 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c: acquiring CA information for follower catalog manager: success
I20260929 03:15:09.621343 11640 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c331a0fd376b482f8edca6fab6d41d8c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:15:09.833647 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:09.829232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.20.174.125:38161)
0929 03:15:09.829523 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:09.829528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:09.829659 (+ 131us) server_negotiation.cc:407] Connection header received
0929 03:15:09.829889 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:09.829892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:09.830022 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:09.830152 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:09.830474 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:09.831040 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:09.832003 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:09.832248 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:09.832368 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:09.832733 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:09.833189 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:09.833348 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:09.833410 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:15:09.833474 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":71,"threads_started":1}
I20260929 03:15:09.840934 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:09.837134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.20.174.124:39087)
0929 03:15:09.837389 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:09.837392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:09.837407 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:15:09.837547 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:09.837550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:09.837669 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:09.837779 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:09.838017 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:09.838665 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:09.839511 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:09.839759 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:09.839841 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:09.840107 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:09.840556 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:09.840652 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:09.840706 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 03:15:09.840796 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260929 03:15:09.848965 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:09.844737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.20.174.125:38161)
0929 03:15:09.844858 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:09.844862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:09.845108 (+ 246us) server_negotiation.cc:407] Connection header received
0929 03:15:09.845210 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:09.845216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:09.845273 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:09.845349 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:09.845636 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:09.846414 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:09.847624 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:09.847941 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:09.848022 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:09.848390 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:09.848470 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:09.848527 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:09.848672 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 03:15:09.848755 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:15:09.849012 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:09.844195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56458 (local address 127.20.174.124:39087)
0929 03:15:09.844315 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:09.844318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:09.844743 (+ 425us) server_negotiation.cc:407] Connection header received
0929 03:15:09.844796 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:09.844799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:09.844865 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:09.844937 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:09.845297 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:09.845941 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:09.847128 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:09.847460 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:09.847578 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:09.847848 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:09.847935 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:09.848015 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:09.848788 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 03:15:09.848865 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:15:09.849102 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:09.844093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.20.174.126:36271)
0929 03:15:09.844353 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:09.844360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:09.844605 (+ 245us) server_negotiation.cc:407] Connection header received
0929 03:15:09.844870 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:09.844873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:09.844999 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:09.845111 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:09.845553 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:09.846405 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:09.847404 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:09.847742 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:09.847841 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:09.848244 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:09.848645 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:09.848734 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:09.848872 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 03:15:09.848943 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":84,"threads_started":1}
I20260929 03:15:09.852862 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 11370
I20260929 03:15:09.859650 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 11445
I20260929 03:15:09.866183 21177 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEa40KE/build/debug/bin/kudu with pid 11519
2026-09-29T03:15:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3247 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6532 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (296354 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST